<<< Date Index >>>     <<< Thread Index >>>

[Mutt] #2981: header_cache: opening a changed Maildir takes way longer than it should



#2981: header_cache: opening a changed Maildir takes way longer than it
        should

 {{{
 ----- Forwarded message from martin f krafft <madduck@xxxxxxxxxx> -----

 Date: Tue, 24 Jul 2007 21:11:33 +0200
 From: martin f krafft <madduck@xxxxxxxxxx>
 Subject: Bug#434544: header_cache: opening a changed Maildir takes way
         longer than it should

 Package: mutt
 Version: 1.5.16-3
 Severity: normal

 If I open mutt on a Maildir it has cached, it's quite quick.
 However, if procmail or postfix write a few messages to the Maildir
 while mutt was not running and mutt is now started, it seems as if
 it re-enumerates *all* messages. In fact, there is hardly
 a noticeable difference in the time it takes to open a previously
 indexed but out-of-date Maildir with or without the header_cache
 parameter turned on; sometimes it's even faster to open the Maildir
 without
 header_cache enabled. Turning the verify parameter off had no effect.

 It gets even worse if messages are written to the Maildir while mutt
 is loading, then it gets even worse.

 The following is done on a system with minimal load, a lot of RAM, ext3
 filesystems on RAID10, and a 3GHz amd64. I'll be happy to share the
 150'000 test body with you.

 piper:~/.tmp/hcache> ls -l
 total 4
 drwx------ 5 madduck madduck 4096 2004-04-17 13:02 large/
 piper:~/.tmp/hcache> echo large/{cur,new}/* | wc -w
 155815
 piper:~/.tmp/hcache> mkdir saved
 piper:~/.tmp/hcache> echo large/{cur,new}/?????????[02468]* | xargs
 -n10000 mv --target-directory=saved
 piper:~/.tmp/hcache> echo large/{cur,new}/* | wc -w
 78365

 ## twice without hc to adjust caches
 piper:~/.tmp/hcache> time mutt -f large -e "set header_cache = ''" -e
 'exec exit'
 9.44s user 6.66s system 15% cpu 1:41.43 total
 piper:~/.tmp/hcache> time mutt -f large -e "set header_cache = ''" -e
 'exec exit'
 9.34s user 6.81s system 18% cpu 1:29.21 total
 piper:~/.tmp/hcache> time mutt -f large -e "set header_cache = ''" -e
 'exec exit'

 ## now create hc and test it
 piper:~/.tmp/hcache> mkdir hc
 piper:~/.tmp/hcache> time mutt -f large -e "set header_cache = `pwd`/hc"
 -e 'exec exit'
 10.66s user 9.54s system 16% cpu 2:03.64 total
 piper:~/.tmp/hcache> time mutt -f large -e "set header_cache = `pwd`/hc"
 -e 'exec exit'
 2.61s user 0.73s system 65% cpu 5.088 total

 ## add some messages
 piper:~/.tmp/hcache> echo saved/????????0* | xargs -n10000 mv --target-
 directory=large/cur
 piper:~/.tmp/hcache> echo large/{cur,new}/* | wc -w
 85280
 piper:~/.tmp/hcache> time mutt -f large -e "set header_cache = `pwd`/hc"
 -e 'exec exit'
 7.16s user 5.38s system 11% cpu 1:49.62 total
 piper:~/.tmp/hcache> time mutt -f large -e "set header_cache = ''" -e
 'exec exit'
 10.24s user 7.04s system 18% cpu 1:32.91 total
 ##### COMPARE THOSE!

 ## the other way around, because of filesystem/disk caches
 piper:~/.tmp/hcache> echo saved/????????3* | xargs -n10000 mv --target-
 directory=large/cur
 piper:~/.tmp/hcache> echo large/{cur,new}/* | wc -w
 93991
 piper:~/.tmp/hcache> time mutt -f large -e "set header_cache = ''" -e
 'exec exit'
 11.00s user 8.22s system 18% cpu 1:43.29 total
 piper:~/.tmp/hcache> time mutt -f large -e "set header_cache = `pwd`/hc"
 -e 'exec exit'
 8.05s user 5.56s system 11% cpu 1:42.17 total

 So while the disk/filesystem caches hardly make a difference, they
 are noticeable.

 I did the same run with maildir_header_cache_verify set to no as
 well, but since I am actually being Maildir-aware by moving files
 back into the Maildir which were previously there, the difference is
 negligible.

 When header_cache is in use, mutt starts and says "Reading large..." and
 quickly counts from 1 to the number of messages. Then, suddenly it just
 starts
 from the beginning and counts much much slower.

 Somehow I don't think this needs to be two-pass. As it stands,
 header_cache actually seems to slow down mutt if the Maildir was
 substantially changed. That really should *not* be.

 -- System Information:
 Debian Release: lenny/sid
   APT prefers unstable
   APT policy: (500, 'unstable'), (500, 'testing'), (500, 'stable'), (1,
 'experimental')
 Architecture: amd64 (x86_64)

 Kernel: Linux 2.6.21-2-amd64 (SMP w/1 CPU core)
 Locale: LANG=en_GB, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8)
 Shell: /bin/sh linked to /bin/dash

 Versions of packages mutt depends on:
 ii  libc6                    2.6-3           GNU C Library: Shared
 libraries
 ii  libgdbm3                 1.8.3-3         GNU dbm database routines
 (runtime
 ii  libgnutls13              1.6.3-1         the GNU TLS library - runtime
 libr
 ii  libidn11                 0.6.5-1         GNU libidn library,
 implementation
 ii  libncursesw5             5.6+20070716-1  Shared libraries for terminal
 hand
 ii  libsasl2-2               2.1.22.dfsg1-13 Authentication abstraction
 library
 }}}

-- 
Ticket URL: <http://dev.mutt.org/trac/ticket/2981>