From: Eleanor McHugh <eleanor@...>
Date: 2009-06-18T04:29:38+09:00
Subject: Re: reading large file in chunks: optimal chunk size?

On 17 Jun 2009, at 00:22, Bill Kelly wrote:
> I'd recommend trying some benchmarks using various
> chunk sizes, and also trying lower level unbuffered
> read (sysread).
>
> Last time I benchmarked this, a few years ago (although
> using C not Ruby) on Win32 (NTFS) and OS X (HFS+, i think)
> I was surprised to find the optimal read chunk size was 4K,
> which happened to be the partition allocation unit size,
> and also the VM page size.
>
> I tried all sorts of chunk sizes.  The result was counter-
> intuitive to me.  I figured, if I allocated a large buffer,
> and made a single read() call, that should be faster, if not _at  
> least as fast_ as making a whole lot of separate
> 4K reads.
>
> But no, 4K was always the fastest in my tests.

Which makes perfect sense when you consider that modern operating  
systems maintain file caches in virtual memory, so once you start  
accessing a file it's going to be mapped into VM and subsequent reads/ 
writes will generate page faults in the kernel and cause one or more  
page-sized chunks to be physically loaded into RAM. That's the point  
at which drive geometry is going to matter so a larger read might  
necessitate several disk accesses to load physically discontinuous but  
logically adjacent blocks.

Once the pages are loaded into the cache all subsequent reads will be  
at RAM speeds rather than HDD speeds and reproducible benchmarks can  
be difficult to harvest at that point: neither process scheduling nor  
in-kernel page loading are deterministic.


Ellie

Eleanor McHugh
Games With Brains
http://slides.games-with-brains.net
----
raise ArgumentError unless @reality.responds_to? :reason