From: pjb@... (Pascal J. Bourguignon) Date: 2009-06-17T18:35:06+09:00 Subject: Re: reading large file in chunks: optimal chunk size? bwv549 writes: > I'm reading a large file (too big to fit in memory) and doing a > hexdigest on it. What are some optimal size chunks to read the file > in and why? (speed is probably most important here as long as enough > memory is available for most machines) Theorically, the fastest I/O could occur when the buffer is page-aligned, and when its size is a multiple of the system block size. If the file was stored continuously on the hard disk, the fastest I/O throughput would be attained by reading cylinder by cylinder. (There could even not be any latency then, since when you read a whole track you don't need to start from the start, you can start in the middle, and loop over). Unfortunately nowadays it would be near impossible to do that, since the hard disk firmware hides the physical layout of the sectors, and may use replacement sectors needing even seeks while reading a single track. The OS file system may also try to spread the file blocks all over the disk (or at least, all over a cylinder group), to avoid having to do long seeks over big files when acccessing small files. So instead of considering physical tracks, you may try to take into account the hard disk buffer size. Most hard disks have buffer size of 8 MB, some 16 MB. So reading the file by chunks of up to 1 MB or 2 MB should let the hard disk firmware optimize the throughput without stalling it's buffer. But then, if there are a lot of layers above, eg, a RAID, all bets are off. Really, the best you can do is to benchmark your particular circumstances. -- __Pascal Bourguignon__