Re: [Mutt] #2981: opening a Maildir takes forever
#2981: opening a Maildir takes forever
-----------------------------------------------+----------------------------
Reporter: Christoph Berg <myon@xxxxxxxxxx> | Owner: mutt-dev
Type: defect | Status: new
Priority: major | Milestone: 2.0
Component: mutt | Version:
Resolution: | Keywords: header cache,
performance, Maildir, sorting
-----------------------------------------------+----------------------------
Old description:
> {{{
> ----- Forwarded message from martin f krafft <madduck@xxxxxxxxxx> -----
>
> Date: Tue, 24 Jul 2007 21:11:33 +0200
> From: martin f krafft <madduck@xxxxxxxxxx>
> Subject: Bug#434544: header_cache: opening a changed Maildir takes way
> longer than it should
>
> Package: mutt
> Version: 1.5.16-3
> Severity: normal
>
> If I open mutt on a Maildir it has cached, it's quite quick.
> However, if procmail or postfix write a few messages to the Maildir
> while mutt was not running and mutt is now started, it seems as if
> it re-enumerates *all* messages. In fact, there is hardly
> a noticeable difference in the time it takes to open a previously
> indexed but out-of-date Maildir with or without the header_cache
> parameter turned on; sometimes it's even faster to open the Maildir
> without
> header_cache enabled. Turning the verify parameter off had no effect.
>
> It gets even worse if messages are written to the Maildir while mutt
> is loading, then it gets even worse.
>
> The following is done on a system with minimal load, a lot of RAM, ext3
> filesystems on RAID10, and a 3GHz amd64. I'll be happy to share the
> 150'000 test body with you.
>
> piper:~/.tmp/hcache> ls -l
> total 4
> drwx------ 5 madduck madduck 4096 2004-04-17 13:02 large/
> piper:~/.tmp/hcache> echo large/{cur,new}/* | wc -w
> 155815
> piper:~/.tmp/hcache> mkdir saved
> piper:~/.tmp/hcache> echo large/{cur,new}/?????????[02468]* | xargs
> -n10000 mv --target-directory=saved
> piper:~/.tmp/hcache> echo large/{cur,new}/* | wc -w
> 78365
>
> ## twice without hc to adjust caches
> piper:~/.tmp/hcache> time mutt -f large -e "set header_cache = ''" -e
> 'exec exit'
> 9.44s user 6.66s system 15% cpu 1:41.43 total
> piper:~/.tmp/hcache> time mutt -f large -e "set header_cache = ''" -e
> 'exec exit'
> 9.34s user 6.81s system 18% cpu 1:29.21 total
> piper:~/.tmp/hcache> time mutt -f large -e "set header_cache = ''" -e
> 'exec exit'
>
> ## now create hc and test it
> piper:~/.tmp/hcache> mkdir hc
> piper:~/.tmp/hcache> time mutt -f large -e "set header_cache = `pwd`/hc"
> -e 'exec exit'
> 10.66s user 9.54s system 16% cpu 2:03.64 total
> piper:~/.tmp/hcache> time mutt -f large -e "set header_cache = `pwd`/hc"
> -e 'exec exit'
> 2.61s user 0.73s system 65% cpu 5.088 total
>
> ## add some messages
> piper:~/.tmp/hcache> echo saved/????????0* | xargs -n10000 mv --target-
> directory=large/cur
> piper:~/.tmp/hcache> echo large/{cur,new}/* | wc -w
> 85280
> piper:~/.tmp/hcache> time mutt -f large -e "set header_cache = `pwd`/hc"
> -e 'exec exit'
> 7.16s user 5.38s system 11% cpu 1:49.62 total
> piper:~/.tmp/hcache> time mutt -f large -e "set header_cache = ''" -e
> 'exec exit'
> 10.24s user 7.04s system 18% cpu 1:32.91 total
> ##### COMPARE THOSE!
>
> ## the other way around, because of filesystem/disk caches
> piper:~/.tmp/hcache> echo saved/????????3* | xargs -n10000 mv --target-
> directory=large/cur
> piper:~/.tmp/hcache> echo large/{cur,new}/* | wc -w
> 93991
> piper:~/.tmp/hcache> time mutt -f large -e "set header_cache = ''" -e
> 'exec exit'
> 11.00s user 8.22s system 18% cpu 1:43.29 total
> piper:~/.tmp/hcache> time mutt -f large -e "set header_cache = `pwd`/hc"
> -e 'exec exit'
> 8.05s user 5.56s system 11% cpu 1:42.17 total
>
> So while the disk/filesystem caches hardly make a difference, they
> are noticeable.
>
> I did the same run with maildir_header_cache_verify set to no as
> well, but since I am actually being Maildir-aware by moving files
> back into the Maildir which were previously there, the difference is
> negligible.
>
> When header_cache is in use, mutt starts and says "Reading large..." and
> quickly counts from 1 to the number of messages. Then, suddenly it just
> starts
> from the beginning and counts much much slower.
>
> Somehow I don't think this needs to be two-pass. As it stands,
> header_cache actually seems to slow down mutt if the Maildir was
> substantially changed. That really should *not* be.
>
> -- System Information:
> Debian Release: lenny/sid
> APT prefers unstable
> APT policy: (500, 'unstable'), (500, 'testing'), (500, 'stable'), (1,
> 'experimental')
> Architecture: amd64 (x86_64)
>
> Kernel: Linux 2.6.21-2-amd64 (SMP w/1 CPU core)
> Locale: LANG=en_GB, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8)
> Shell: /bin/sh linked to /bin/dash
>
> Versions of packages mutt depends on:
> ii libc6 2.6-3 GNU C Library: Shared
> libraries
> ii libgdbm3 1.8.3-3 GNU dbm database routines
> (runtime
> ii libgnutls13 1.6.3-1 the GNU TLS library -
> runtime libr
> ii libidn11 0.6.5-1 GNU libidn library,
> implementation
> ii libncursesw5 5.6+20070716-1 Shared libraries for
> terminal hand
> ii libsasl2-2 2.1.22.dfsg1-13 Authentication abstraction
> library
> }}}
New description:
{{{
----- Forwarded message from martin f krafft <madduck@xxxxxxxxxx> -----
Date: Tue, 24 Jul 2007 21:11:33 +0200
From: martin f krafft <madduck@xxxxxxxxxx>
Subject: Bug#434544: header_cache: opening a changed Maildir takes way
longer than it should
Package: mutt
Version: 1.5.16-3
Severity: normal
If I open mutt on a Maildir it has cached, it's quite quick.
However, if procmail or postfix write a few messages to the Maildir
while mutt was not running and mutt is now started, it seems as if
it re-enumerates *all* messages. In fact, there is hardly
a noticeable difference in the time it takes to open a previously
indexed but out-of-date Maildir with or without the header_cache
parameter turned on; sometimes it's even faster to open the Maildir
without
header_cache enabled. Turning the verify parameter off had no effect.
It gets even worse if messages are written to the Maildir while mutt
is loading, then it gets even worse.
The following is done on a system with minimal load, a lot of RAM, ext3
filesystems on RAID10, and a 3GHz amd64. I'll be happy to share the
150'000 test body with you.
piper:~/.tmp/hcache> ls -l
total 4
drwx------ 5 madduck madduck 4096 2004-04-17 13:02 large/
piper:~/.tmp/hcache> echo large/{cur,new}/* | wc -w
155815
piper:~/.tmp/hcache> mkdir saved
piper:~/.tmp/hcache> echo large/{cur,new}/?????????[02468]* | xargs
-n10000 mv --target-directory=saved
piper:~/.tmp/hcache> echo large/{cur,new}/* | wc -w
78365
## twice without hc to adjust caches
piper:~/.tmp/hcache> time mutt -f large -e "set header_cache = ''" -e
'exec exit'
9.44s user 6.66s system 15% cpu 1:41.43 total
piper:~/.tmp/hcache> time mutt -f large -e "set header_cache = ''" -e
'exec exit'
9.34s user 6.81s system 18% cpu 1:29.21 total
piper:~/.tmp/hcache> time mutt -f large -e "set header_cache = ''" -e
'exec exit'
## now create hc and test it
piper:~/.tmp/hcache> mkdir hc
piper:~/.tmp/hcache> time mutt -f large -e "set header_cache = `pwd`/hc"
-e 'exec exit'
10.66s user 9.54s system 16% cpu 2:03.64 total
piper:~/.tmp/hcache> time mutt -f large -e "set header_cache = `pwd`/hc"
-e 'exec exit'
2.61s user 0.73s system 65% cpu 5.088 total
## add some messages
piper:~/.tmp/hcache> echo saved/????????0* | xargs -n10000 mv --target-
directory=large/cur
piper:~/.tmp/hcache> echo large/{cur,new}/* | wc -w
85280
piper:~/.tmp/hcache> time mutt -f large -e "set header_cache = `pwd`/hc"
-e 'exec exit'
7.16s user 5.38s system 11% cpu 1:49.62 total
piper:~/.tmp/hcache> time mutt -f large -e "set header_cache = ''" -e
'exec exit'
10.24s user 7.04s system 18% cpu 1:32.91 total
##### COMPARE THOSE!
## the other way around, because of filesystem/disk caches
piper:~/.tmp/hcache> echo saved/????????3* | xargs -n10000 mv --target-
directory=large/cur
piper:~/.tmp/hcache> echo large/{cur,new}/* | wc -w
93991
piper:~/.tmp/hcache> time mutt -f large -e "set header_cache = ''" -e
'exec exit'
11.00s user 8.22s system 18% cpu 1:43.29 total
piper:~/.tmp/hcache> time mutt -f large -e "set header_cache = `pwd`/hc"
-e 'exec exit'
8.05s user 5.56s system 11% cpu 1:42.17 total
So while the disk/filesystem caches hardly make a difference, they
are noticeable.
I did the same run with maildir_header_cache_verify set to no as
well, but since I am actually being Maildir-aware by moving files
back into the Maildir which were previously there, the difference is
negligible.
When header_cache is in use, mutt starts and says "Reading large..." and
quickly counts from 1 to the number of messages. Then, suddenly it just
starts
from the beginning and counts much much slower.
Somehow I don't think this needs to be two-pass. As it stands,
header_cache actually seems to slow down mutt if the Maildir was
substantially changed. That really should *not* be.
-- System Information:
Debian Release: lenny/sid
APT prefers unstable
APT policy: (500, 'unstable'), (500, 'testing'), (500, 'stable'), (1,
'experimental')
Architecture: amd64 (x86_64)
Kernel: Linux 2.6.21-2-amd64 (SMP w/1 CPU core)
Locale: LANG=en_GB, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Versions of packages mutt depends on:
ii libc6 2.6-3 GNU C Library: Shared
libraries
ii libgdbm3 1.8.3-3 GNU dbm database routines
(runtime
ii libgnutls13 1.6.3-1 the GNU TLS library - runtime
libr
ii libidn11 0.6.5-1 GNU libidn library,
implementation
ii libncursesw5 5.6+20070716-1 Shared libraries for terminal
hand
ii libsasl2-2 2.1.22.dfsg1-13 Authentication abstraction
library
}}}
--
Comment(by pdmef):
Also see ticket #3166. Can you please see if changeset [eb918af802ec]
fixed the issue?
--
Ticket URL: <http://dev.mutt.org/trac/ticket/2981#comment:8>
Mutt <http://www.mutt.org/>
The Mutt mail user agent