Re: fadvise WILLNEED for tokyocabinet header cache
Hi,
* Andrea Arcangeli wrote:
> On Sun, Jun 21, 2009 at 11:05:28PM +0200, Rocco Rutte wrote:
> > points that would have to be solved to get the patch ready for
> > inclusion. For example test if posix_fadvise() is available, why only
>
> That requires a configure knob?
No, simply test for its existence, and wrap the code in hcache.c in
#ifdef/#endif. There's no need for a knob.
> > do that for tokyocabinet, etc. Note that we're in a feature-freeze to
>
> No good reason to do it only for tokyocabinet indeed. But if one wants
> performance he should use something with compression so that one reads
> 1/5th of the data at 3 times the speed with fadvise than without
> fadvise.
Well, as tokyocabinet is the successor of qdbm, qdbm has compression,
too. I don't know about DB4, but the other 3 should have that function
even if the benefit is only a few percent.
> Cumulatively compression and fadvise reduce the opening time
> tenfold, ignoring the time it actually takes to process the data by
> the db lib after the I/O is complete. I agree we should add fadvise to
> all backends even if it makes little sense to use anything but
> tokyocabinet.
Care to provide some details why it makes no sense to use qdbm? :-)
> > It's there because mutt displays what the mailbox has, not what the
> > cache has. The cur/ directory may change greatly between to mutt
> > sessions (e.g. a sync using unison, rsync, etc). I can't imagine
> > dropping it (except for a replacement mechanism that tells us what files
> > there are).
> What if I can guarantee you that I don't modify the 'cur' directory
> under 'mutt'. The only thing that touches my maildir are procmail in
> 'tmp' and 'new' and mutt in 'cur' and 'new'. Nothing else ever touches
> my maildir, so I definitely need a knob to disable readdir on 'cur'
> and to only have readdir on 'new' when opening a maildir so with the
> addition of fadvise and tokyocabinet compression things will fly even
> for large folders with 150k mails ;).
Yes, I see your point. The only thing I'm sure is that your guarantee
shouldn't be generalized. :) What not sure about is whether we really
want an option for it. Hacking this isn't exactly trivial either given
that the current code of two pass maildir parsing.
Rocco