Re: reading large file in chunks: optimal chunk size?

bwv549 <[e-mail]> writes:

> I'm reading a large file (too big to fit in memory) and doing a
> hexdigest on it.  What are some optimal size chunks to read the file
> in and why?  (speed is probably most important here as long as enough
> memory is available for most machines)

Theorically, the fastest I/O could occur when the buffer is
page-aligned, and when its size is a multiple of the system block
size.


If the file was stored continuously on the hard disk, the fastest I/O
throughput would be attained by reading cylinder by cylinder.  (There
could even not be any latency then, since when you read a  whole track
you don't need to start from the start, you can start in the middle,
and loop over).  Unfortunately nowadays it would be near impossible to
do that, since the hard disk firmware hides the physical layout of the
sectors, and may use replacement sectors needing even seeks while
reading a single track.  The OS file system may also try to spread the
file blocks all over the disk (or at least, all over a cylinder
group), to avoid having to do long seeks over big files when
acccessing small files.  

So instead of considering physical tracks, you may try to take into
account the hard disk buffer size. Most hard disks have buffer size of
8 MB, some 16 MB.  So reading the file by chunks of up to 1 MB or 2 MB
should let the hard disk firmware optimize the throughput without
stalling it's buffer.

But then, if there are a lot of layers above, eg, a RAID, all bets are
off.

Really, the best you can do is to benchmark your particular
circumstances.

-- 
__Pascal Bourguignon__