Re: mutt - slow mbox'es
* David Yitzchak Cohen <lists+mutt_users@xxxxxxxxxxxxxx> [2004-07-22 12:50
-0400]:
> On Thu, Jul 22, 2004 at 10:11:41AM EDT, Nicolas Rachinsky wrote:
> > The following commands were executed on a mbox and maildir containing
> > the same 22959 mails with a total size of 300MB.
>
> You've got fat emails. My inbox is well over 50K emails. If you're
> gonna enter an ext2 directory with that many files, I'd suggest at
> least grabbing a Snickers bar ("Not going anywhere for a while? Grab a
> Snickers :-)").
Yes, there are some (many) fat mails in this testbox. I'll try again
with some smaller mails.
> Even with your own box, though, notice how much time is wasted (a)
> in systemcalls and (b) in actual disk I/O. The second open, BTW,
> wouldn't be half as fast if you didn't have 1GB of RAM. (Try 192MB,
> and then we can talk ... hehe. . .)
Can't try this easily, sorry.
> Your benchmarks are quoted below in their entirety, for ease of reference.
> > The following commands were executed on a mbox and maildir containing
> > the same 22959 mails with a total size of 300MB.
> >
> > /usr/bin/time -h -l mutt -F /dev/null -f big.mbox -e 'push x'
> > 10.01s real 2.47s user 0.65s sys
> > 2263 block input operations
> > 0 block output operations
> >
> > /usr/bin/time -h -l mutt -F /dev/null -f big.mbox -e 'push x'
> > 3.12s real 2.42s user 0.35s sys
> > 0 block input operations
> > 0 block output operations
> >
> >
> >
> > /usr/bin/time -h -l mutt -F /dev/null -f big.maildir -e 'push x'
> > 19.51s real 2.42s user 1.99s sys
> > 23598 block input operations
> > 0 block output operations
> >
> > /usr/bin/time -h -l mutt -F /dev/null -f big.maildir -e 'push x'
> > 3.46s real 2.28s user 0.86s sys
> > 0 block input operations
> > 0 block output operations
> >
> >
> >
> > /usr/bin/time -h -l mutt -F /dev/null -f big.maildir -e 'set
> > maildir_cache=cache' -e 'push x'
> > 6.54s real 3.16s user 1.49s sys
> > 2 block input operations
> > 744 block output operations
> >
> >
> >
> > /usr/bin/time -h -l mutt -F /dev/null -f big.maildir -e 'set
> > maildir_cache=cache' -e 'push x'
> > 1.43s real 0.79s user 0.36s sys
> > 0 block input operations
> > 12 block output operations
> >
> >
> > /usr/bin/time -h -l mutt -F /dev/null -f big.maildir -e 'set
> > maildir_cache=cache' -e 'push x'
> > 1.40s real 0.79s user 0.37s sys
> > 0 block input operations
> > 12 block output operations
> >
> >
> >
> > /usr/bin/time -h -l mutt -F /dev/null -f big.maildir -e 'set
> > maildir_cache=cache' -e 'unset maildir_cache_verify' -e 'push x'
> > 1.20s real 0.72s user 0.25s sys
> > 0 block input operations
> > 12 block output operations
> >
> >
> > I removed all except the interessting lines from the time output.
After removing cache the same with another Mailbox containing now
68244 emails with a total size of 300MB.
/usr/bin/time -h -l mutt -F /dev/null -f small.mbox -e 'push x'
25.82s real 14.87s user 1.81s sys
4859 block input operations
0 block output operations
/usr/bin/time -h -l mutt -F /dev/null -f small.mbox -e 'push x'
16.91s real 14.63s user 1.16s sys
0 block input operations
0 block output operations
/usr/bin/time -h -l mutt -F /dev/null -f small.maildir -e 'push x'
39.63s real 8.73s user 7.21s sys
68963 block input operations
0 block output operations
/usr/bin/time -h -l mutt -F /dev/null -f small.maildir -e 'push x'
36.66s real 8.61s user 6.52s sys
68253 block input operations
0 block output operations
/usr/bin/time -h -l mutt -F /dev/null -f small.maildir -e 'set
maildir_cache=cache' -e 'push x'
49.13s real 11.19s user 8.64s sys
68260 block input operations
644 block output operations
/usr/bin/time -h -l mutt -F /dev/null -f small.maildir -e 'set
maildir_cache=cache' -e 'push x'
4.49s real 2.48s user 1.36s sys
5 block input operations
12 block output operations
/usr/bin/time -h -l mutt -F /dev/null -f small.maildir -e 'set
maildir_cache=cache' -e 'push x'
4.45s real 2.46s user 1.39s sys
2 block input operations
12 block output operations
/usr/bin/time -h -l mutt -F /dev/null -f small.maildir -e 'set
maildir_cache=cache' -e 'unset maildir_cache_verify' -e 'push x'
3.87s real 2.35s user 0.84s sys
1 block input operations
12 block output operations
BTW:
136M cache
And now a third folder, containing 18 emails with a total size of 310MB.
/usr/bin/time -h -l mutt -F /dev/null -f monster.mbox -e 'push x'
0.40s real 0.01s user 0.01s sys
47 block input operations
0 block output operations
/usr/bin/time -h -l mutt -F /dev/null -f monster.mbox -e 'push x'
0.03s real 0.01s user 0.01s sys
0 block input operations
0 block output operations
/usr/bin/time -h -l mutt -F /dev/null -f monster.maildir -e 'push x'
0.16s real 0.01s user 0.00s sys
39 block input operations
0 block output operations
/usr/bin/time -h -l mutt -F /dev/null -f monster.maildir -e 'push x'
0.02s real 0.01s user 0.00s sys
0 block input operations
0 block output operations
/usr/bin/time -h -l mutt -F /dev/null -f monster.maildir -e 'set
maildir_cache=cache' -e 'push x'
0.05s real 0.01s user 0.01s sys
1 block input operations
25 block output operations
/usr/bin/time -h -l mutt -F /dev/null -f monster.maildir -e 'set
maildir_cache=cache' -e 'push x'
0.02s real 0.01s user 0.00s sys
0 block input operations
12 block output operations
/usr/bin/time -h -l mutt -F /dev/null -f monster.maildir -e 'set
maildir_cache=cache' -e 'push x'
0.03s real 0.01s user 0.00s sys
0 block input operations
12 block output operations
/usr/bin/time -h -l mutt -F /dev/null -f monster.maildir -e 'set
maildir_cache=cache' -e 'unset maildir_cache_verify' -e 'push x'
0.02s real 0.01s user 0.00s sys
0 block input operations
12 block output operations
The same mbox, but with Content-Length Headers removed:
/usr/bin/time -h -l mutt -F /dev/null -f monster2 -e 'push x'
13.06s real 1.33s user 1.16s sys
5001 block input operations
0 block output operations
/usr/bin/time -h -l mutt -F /dev/null -f monster2 -e 'push x'
2.04s real 1.27s user 0.75s sys
0 block input operations
0 block output operations
Nicolas
> > PS:
> > Mutt 1.5.6i (2004-02-01)
> > Copyright (C) 1996-2002 Michael R. Elkins and others.
> > Mutt comes with ABSOLUTELY NO WARRANTY; for details type `mutt -vv'.
> > Mutt is free software, and you are welcome to redistribute it
> > under certain conditions; type `mutt -vv' for details.
> >
> > System: FreeBSD 4.10-RELEASE-p2 (i386) [using ncurses 5.1] [using libiconv
> > 1.9] [using libidn 0.5.2 (compiled with 0.5.2)]
> > Compile options:
> > -DOMAIN
> > -DEBUG
> > -HOMESPOOL +USE_SETGID +USE_DOTLOCK +DL_STANDALONE
> > -USE_FCNTL +USE_FLOCK
> > +USE_POP +USE_IMAP +IMAP_EDIT_THREADS -USE_GSS +USE_SSL -USE_SASL
> > -USE_SASL2
> > +HAVE_REGCOMP -USE_GNU_REGEX +COMPRESSED
> > +HAVE_COLOR +HAVE_START_COLOR +HAVE_TYPEAHEAD +HAVE_BKGDSET
> > +HAVE_CURS_SET +HAVE_META +HAVE_RESIZETERM
> > +CRYPT_BACKEND_CLASSIC_PGP +CRYPT_BACKEND_CLASSIC_SMIME
> > -CRYPT_BACKEND_GPGME -BUFFY_SIZE -EXACT_ADDRESS -SUN_ATTACHMENT
> > +ENABLE_NLS -LOCALES_HACK -HAVE_WC_FUNCS +HAVE_LANGINFO_CODESET
> > +HAVE_LANGINFO_YESEXPR
> > +HAVE_ICONV -ICONV_NONTRANS +HAVE_LIBIDN +HAVE_GETSID +HAVE_GETADDRINFO
> > +USE_HCACHE
> > -ISPELL
> > SENDMAIL="/usr/sbin/sendmail"
> > MAILPATH="/var/mail"
> > PKGDATADIR="/usr/local/share/mutt"
> > SYSCONFDIR="/usr/local/etc"
> > EXECSHELL="/bin/sh"
> > -MIXMASTER
> > To contact the developers, please mail to <mutt-dev@xxxxxxxx>.
> > To report a bug, please use the flea(1) utility.
> >
> > patch-1.5.6.dw.multiple-crypt-hook.2
> > patch-1.5.6.dw.confirm-crypt-hook.1
> > patch-1.5.6.dw.crypt-autoselectkey.1
> > patch-1.5.6.dw.pgp-menu-traditional.2
> > patch-1.5.6.dw.maildir-mtime.1
> > patch-1.5.6.dshaw.extend_timeout
> > maildir-header-cache.18
> > patch-1.5.6.eh.thread_subject.1
> > patch-1.5.5.1.cd.trash_folder.3.4
> > patch-1.5.1.nr.indicator_not_bright
> > 1.3.26.bj.current_shortcut.1-nr
> > rr.compressed
> > 1.3.28.nr.threadcomplete
> > patch-1.5.0.ats.date_conditional.1
> > dgc.deepif.1
> >
> > -----
> >
> > FreeBSD pc5.i.0x5.de 4.10-RELEASE-p2 FreeBSD 4.10-RELEASE-p2 #0: Fri Jul 2
> > 09:03:30 CEST 2004
> > nicolas@xxxxxxxxxxxx:/scratch/obj/usr/exported/src/sys/PC5 i386
> >
> > -----
> >
> > CPU: AMD Athlon(TM) XP 2000+ (1666.74-MHz 686-class CPU)
> >
> > 1GB RAM
> >
> > -----
> >
> > FFS with softupdates on a Vinum RAID1 on
> > ad0: 152627MB <SAMSUNG SV1604N> [310101/16/63] at ata0-master UDMA100
> > ad6: 156334MB <Maxtor 6Y160P0> [317632/16/63] at ata3-master UDMA133