From: Bill Kelly <billk@...>
Date: 2009-06-17T08:22:18+09:00
Subject: Re: reading large file in chunks: optimal chunk size?


From: "bwv549" <jtprince@gmail.com>
>
> I'm reading a large file (too big to fit in memory) and doing a
> hexdigest on it.  What are some optimal size chunks to read the file
> in and why?  (speed is probably most important here as long as enough
> memory is available for most machines)

I'd recommend trying some benchmarks using various
chunk sizes, and also trying lower level unbuffered
read (sysread).

Last time I benchmarked this, a few years ago (although
using C not Ruby) on Win32 (NTFS) and OS X (HFS+, i think)
I was surprised to find the optimal read chunk size was 4K,
which happened to be the partition allocation unit size,
and also the VM page size.

I tried all sorts of chunk sizes.  The result was counter-
intuitive to me.  I figured, if I allocated a large buffer,
and made a single read() call, that should be faster, if 
not _at least as fast_ as making a whole lot of separate
4K reads.

But no, 4K was always the fastest in my tests.

But, maybe it will be different for you, so if it's 
important, just benchmark it.  :)


Regards,

Bill