From: pp <tdihp@...>
Date: 2011-02-24T17:15:42+09:00
Subject: Re: Fast alternatives to "File" and "IO" for large numbers of files ?

--_9cf3d47b-98aa-470c-98a3-b8df01e1f97e_
Content-Type: text/plain; charset="gb2312"
Content-Transfer-Encoding: 8bit


Hi, could you be more specific on what do you do with the small files, read/write in per-line or whole file?for rapid file ops due to file system heaps(or sort) may be slow anyway.so maybe you can try less file ops, for example, write a file with a single string may serve the io cache well. or, maybe, have a lot of files to write/read in a new thread, so that IO may not interfere your none-IO calculations, if you have some

> Date: Thu, 24 Feb 2011 12:09:48 +0900
> From: phil@pricom.com.au
> Subject: Fast alternatives to "File" and "IO" for large numbers of files ?
> To: ruby-talk@ruby-lang.org
> 
> People,
> 
> I have script that does:
> 
> - statistical processing from data in 50x32x20 (32,000) large input files
> 
> - writes a small text file (22 lines with one or more columns of numbers)
> for each input file
> 
> - read all small files back in again for final processing.
> 
> Profiling shows that IO is taking up more than 60% of the time - short of
> making fewer, larger files for the data (which is inconvenient for random
> viewing/ processing of individual results) are there other alternatives to
> using the "File" and "IO" classes that would be faster?
> 
> Thanks,
> 
> Phil.
> 
 		 	   		  
--_9cf3d47b-98aa-470c-98a3-b8df01e1f97e_--