From: Philip Rhoades <phil@...>
Date: 2011-02-24T12:09:48+09:00
Subject: Fast alternatives to "File" and "IO" for large numbers of files ?

People,

I have script that does:

- statistical processing from data in 50x32x20 (32,000) large input files

- writes a small text file (22 lines with one or more columns of numbers)
for each input file

- read all small files back in again for final processing.

Profiling shows that IO is taking up more than 60% of the time - short of
making fewer, larger files for the data (which is inconvenient for random
viewing/ processing of individual results) are there other alternatives to
using the "File" and "IO" classes that would be faster?

Thanks,

Phil.