Re: [Mutt] #1216: mutt unusable on large Maildir with frequent
#1216: mutt unusable on large Maildir with frequent message delivery
Old description:
> {{{
> Package: mutt
> Version: 1.3.28-1
> Severity: important
>
> [NOTE: this bug report has been submitted to the debian BTS as
> Bug#148435.
> Please Cc all your replies to 148435@xxxxxxxxxxxxxxx .]
>
> From: Brian Ristuccia <brian@xxxxxxxxxxxxx>
> Subject: mutt unusable on large Maildir with frequent message delivery
> Date: Tue, 28 May 2002 23:49:30 -0400
>
> On a large Maildir with roughly 40,000 messages and a new message
> arriving
> every 30 seconds, Mutt is mostly unusable. On an Intel Celeron 366 with
> 384MB of ram running Linux kernel 2.4.17, it takes about a minute to read
> the mailbox. By the time Mutt finishes, another message has arrived and
> so
> Mutt starts reading the mailbox again from the beginning.
>
> If this is not a kernel bug, then Mutt has a serious scalability issue.
>
> --
> Brian Ristuccia
> brian@xxxxxxxxxxxxx
> bristucc@xxxxxxxxxx
>
> Received: (at submit) by bugs.guug.de; 2 Dec 2002 01:48:08 +0000
> From merlin@xxxxxxxxxxx Mon Dec 02 02:48:08 2002
> Received: from magic.merlins.org
> ([216.200.201.205] helo=mail1.merlins.org ident=mail)
> by trithemius.gnupg.org with esmtp (Exim 3.35 #1 (Debian))
> id 18Ifgu-0003rG-00
> for <submit@xxxxxxxxxxxx>; Mon, 02 Dec 2002 02:48:08 +0100
> Received: from saroumane.merlins.org ([198.144.206.12]:32900
> helo=gandalf.merlins.org)
> by mail1.merlins.org with asmtp
> (Cipher TLSv1:DES-CBC3-SHA:168) (Exim 4.10-mm1 #1 (Debian))
> id 18Ific-00063Y-00 by authid <gandalf.merlins.org> with
> auth_plain
> for <submit@xxxxxxxxxxxx>; Sun, 01 Dec 2002 17:49:55 -0800
> Received: from merlin by gandalf.merlins.org with local (Exim 4.10-mm1 #1
> (Debian))
> id 18Ifib-0005Pn-00 by authid <merlin>
> for <submit@xxxxxxxxxxxx>; Sun, 01 Dec 2002 17:49:53 -0800
> Date: Sun, 1 Dec 2002 17:49:53 -0800
> From: Marc MERLIN <marc_soft@xxxxxxxxxxx>
> To: submit@xxxxxxxxxxxx
> Subject: Inefficiency of maildir code -> 1.5.0-me.hcache.8 tryout
> Message-ID: <20021202014953.GB18895@xxxxxxxxxxx>
> Mime-Version: 1.0
> Content-Type: text/plain; charset=us-ascii
> Content-Disposition: inline
> User-Agent: Mutt/1.4i
> X-Sysadmin: BOFH
> X-URL: http://marc.merlins.org/
> X-Operating-System: Proudly running Linux 2.4.19-fswan-1.96-evms-1.1.0
> -swsusp-grsec-1.9.7-marc4-up/Debian woody
> X-Mailer: Some Outlooks can't quote properly without this header
>
> Package: mutt
> Version: 1.5
> Severity: normal
>
> mutt completely trashes my machine for several minutes when it rescans a
> Maildir folder with 20-25k messages
>
> 1.5.0-me.hcache.8 helps for the initial scan when mutt is launched, but
> a rescan after mutt has been launched is still just as slow.
>
> Here are details I posted to the list
>
> On Fri, Nov 08, 2002 at 04:59:48PM +0100, Lorens Kockum wrote:
> > Wouldn't this also be affected by the maildir header caching
> > patch?
>
> Ok, let me report back.
> I have a 24,000 message maildir folder, which isn't small, but isn't
> *that* huge either (especially for maildir)
> I tried mutt 1.4 vs mutt 1.5.0 vs mutt 1.5.0-me.hcache.8.
> The tests were:
> - mutt -f folder, and wait for the counter to reach 24,000
> - wait some more until I can see the index (instant on mutt 1.4)
> - select a message, add with another mutt a message to the opened
> maildir, and type 'i' in the test mutt to get back to the index.
>
> Note that some of those tests in real life on my laptop (800Mhz, 256MB
> Ram, 40G (5400 RPM)) take 4, 5mn or more when everything isn't cached by
> the OS already. I ran these tests a few times until the number had been
> lowered enough and my hard drive light pretty much stayed off.
>
> Times are from t=0, i.e. are not cumulative
>
> mutt 1.4 mutt 1.5.0 mutt
> 1.5.0-me.hcache.8
> test #1 test #2
> scanning 24000 messages 30 sec 10 sec 10 sec 8 sec
> giving the index 30 sec 30 sec 120 sec 10 sec
> rescanning maildir 180 sec 90 sec 90 sec 90 sec
>
> So, as you can see, a maildir rescan after I add a message to an open
> folder is still unacceptably slow, although it's indeed twice as fast in
> mutt 1.5.0
>
> mutt 1.5.0-me.hcache.8 took a long time to open my folder the first
> time, but that's because it was building the index. The following times,
> it opened it in a record 8 to 10 seconds
>
> So Michael's new code makes a great difference for opening a maildir
> folder once the index has been built, unfortunately, it still takes 90
> secs, or 3 times what it would take to kill mutt and restart it, for
> mutt to resync its index with a maildir folder that was modified.
>
> It looks like we're moving in the right direction though.
>
> Let me know if you'd like me to report back on other patches.
>
> Thanks,
> Marc
> --
> "A mouse is a device used to point at the xterm you want to type in" -
> A.S.R.
> Microsoft is to operating systems & security ....
> .... what McDonalds is to gourmet
> cooking
> Home page: http://marc.merlins.org/ | Finger marc_f@xxxxxxxxxxx for
> PGP key
>
> >How-To-Repeat:
>
> >Fix:
> }}}
New description:
{{{
Package: mutt
Version: 1.3.28-1
Severity: important
[NOTE: this bug report has been submitted to the debian BTS as Bug#148435.
Please Cc all your replies to 148435@xxxxxxxxxxxxxxx .]
From: Brian Ristuccia <brian@xxxxxxxxxxxxx>
Subject: mutt unusable on large Maildir with frequent message delivery
Date: Tue, 28 May 2002 23:49:30 -0400
On a large Maildir with roughly 40,000 messages and a new message arriving
every 30 seconds, Mutt is mostly unusable. On an Intel Celeron 366 with
384MB of ram running Linux kernel 2.4.17, it takes about a minute to read
the mailbox. By the time Mutt finishes, another message has arrived and so
Mutt starts reading the mailbox again from the beginning.
If this is not a kernel bug, then Mutt has a serious scalability issue.
--
Brian Ristuccia
brian@xxxxxxxxxxxxx
bristucc@xxxxxxxxxx
Received: (at submit) by bugs.guug.de; 2 Dec 2002 01:48:08 +0000
From merlin@xxxxxxxxxxx Mon Dec 02 02:48:08 2002
Received: from magic.merlins.org
([216.200.201.205] helo=mail1.merlins.org ident=mail)
by trithemius.gnupg.org with esmtp (Exim 3.35 #1 (Debian))
id 18Ifgu-0003rG-00
for <submit@xxxxxxxxxxxx>; Mon, 02 Dec 2002 02:48:08 +0100
Received: from saroumane.merlins.org ([198.144.206.12]:32900
helo=gandalf.merlins.org)
by mail1.merlins.org with asmtp
(Cipher TLSv1:DES-CBC3-SHA:168) (Exim 4.10-mm1 #1 (Debian))
id 18Ific-00063Y-00 by authid <gandalf.merlins.org> with
auth_plain
for <submit@xxxxxxxxxxxx>; Sun, 01 Dec 2002 17:49:55 -0800
Received: from merlin by gandalf.merlins.org with local (Exim 4.10-mm1 #1
(Debian))
id 18Ifib-0005Pn-00 by authid <merlin>
for <submit@xxxxxxxxxxxx>; Sun, 01 Dec 2002 17:49:53 -0800
Date: Sun, 1 Dec 2002 17:49:53 -0800
From: Marc MERLIN <marc_soft@xxxxxxxxxxx>
To: submit@xxxxxxxxxxxx
Subject: Inefficiency of maildir code -> 1.5.0-me.hcache.8 tryout
Message-ID: <20021202014953.GB18895@xxxxxxxxxxx>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.4i
X-Sysadmin: BOFH
X-URL: http://marc.merlins.org/
X-Operating-System: Proudly running Linux 2.4.19-fswan-1.96-evms-1.1.0
-swsusp-grsec-1.9.7-marc4-up/Debian woody
X-Mailer: Some Outlooks can't quote properly without this header
Package: mutt
Version: 1.5
Severity: normal
mutt completely trashes my machine for several minutes when it rescans a
Maildir folder with 20-25k messages
1.5.0-me.hcache.8 helps for the initial scan when mutt is launched, but
a rescan after mutt has been launched is still just as slow.
Here are details I posted to the list
On Fri, Nov 08, 2002 at 04:59:48PM +0100, Lorens Kockum wrote:
> Wouldn't this also be affected by the maildir header caching
> patch?
Ok, let me report back.
I have a 24,000 message maildir folder, which isn't small, but isn't
*that* huge either (especially for maildir)
I tried mutt 1.4 vs mutt 1.5.0 vs mutt 1.5.0-me.hcache.8.
The tests were:
- mutt -f folder, and wait for the counter to reach 24,000
- wait some more until I can see the index (instant on mutt 1.4)
- select a message, add with another mutt a message to the opened
maildir, and type 'i' in the test mutt to get back to the index.
Note that some of those tests in real life on my laptop (800Mhz, 256MB
Ram, 40G (5400 RPM)) take 4, 5mn or more when everything isn't cached by
the OS already. I ran these tests a few times until the number had been
lowered enough and my hard drive light pretty much stayed off.
Times are from t=0, i.e. are not cumulative
mutt 1.4 mutt 1.5.0 mutt
1.5.0-me.hcache.8
test #1 test #2
scanning 24000 messages 30 sec 10 sec 10 sec 8 sec
giving the index 30 sec 30 sec 120 sec 10 sec
rescanning maildir 180 sec 90 sec 90 sec 90 sec
So, as you can see, a maildir rescan after I add a message to an open
folder is still unacceptably slow, although it's indeed twice as fast in
mutt 1.5.0
mutt 1.5.0-me.hcache.8 took a long time to open my folder the first
time, but that's because it was building the index. The following times,
it opened it in a record 8 to 10 seconds
So Michael's new code makes a great difference for opening a maildir
folder once the index has been built, unfortunately, it still takes 90
secs, or 3 times what it would take to kill mutt and restart it, for
mutt to resync its index with a maildir folder that was modified.
It looks like we're moving in the right direction though.
Let me know if you'd like me to report back on other patches.
Thanks,
Marc
--
"A mouse is a device used to point at the xterm you want to type in" -
A.S.R.
Microsoft is to operating systems & security ....
.... what McDonalds is to gourmet
cooking
Home page: http://marc.merlins.org/ | Finger marc_f@xxxxxxxxxxx for
PGP key
>How-To-Repeat:
>Fix:
}}}
Comment (by brendan):
See also #1931
--
Ticket URL: <http://dev.mutt.org/trac/ticket/1216#comment:4>