From: David King Landrith Date: 2003-04-18T01:11:07+09:00 Subject: Re: Need IO Optimization help On Thursday, April 17, 2003, at 09:46 AM, Brian Candler wrote: > On Thu, Apr 17, 2003 at 10:34:53PM +0900, ts wrote: >> Jim Freeze writes: >>> >>> Good luck, for this >>> >>> MMapIO.new(readfile).each { |line >>> # start rockin' >>> f << line >>> } >>> >>> >>> Has this been written before I start to do it myself? >> >> yes, but don't expect that it will be faster ... > > I agree completely, because I've run essentially that program (but > using > Ruby's standard I/O library) through a profiler. It spends around 5% > of its > time doing I/O. > > The problems are: > - you want to process the file a *line* at a time > - you are allocating a new String object for each line > - you are calling 'yield' on a Ruby block for each line I've found that in the real world the speed advantages of using mmap are more significant than you seem to allow for. (see "My experiments" below.) > As a guess, I'd say your savings would be minimal. Should you decide > to mmap > the whole 260MB into your address space at once you may actually get > worse > overall performance, unless you have that amount of free RAM available. This is a real possibility, if not probability. I believe in my original email, I alluded to this as a disadvantage of using mmap. However, it will depend very heavily upon how your system implements mmap, and there are wildly varying implementations. For example, Linux Kernels before 2.4.20 probably don't handle files that size well. Mmap can also have widely varying results on different systems, and with different functions on the same system. I'm using MacOS X, for example, and its mmap is happy with strncpy(x, mmapVar, n) but takes a significant performance hit when you do strncat(x, mmapVar, n); I have no idea why. I do not remember such a problem with Linux. My experiments -------------- I wrote a function as a Ruby extension (Text_CSV.decode) that uses a state machine to parse files one byte at a time and convert each line from a character delimited record to an array. At a high level, it looks like this: Accumulate bytes for a given field into a c string. When field terminator is found, convert c string to ruby string and push it onto a ruby array. When record terminator is found, either yield the array or push a clone of it onto another array. (There are also other issues like quotes, escaped characters, and the ability to filter non-visible ASCII character). Thus, there are a lot of comparisons, conditionals, and strncpy calls from mmap to the field accumulator. Moreover, it tests for whether a block is given and yields a completed row as a ruby array if so. My early versions buffered IO from the the low level open and close commands in unistd.h. I switched to mmap because I was frustrated with the performance. Unfortunately, I don't have the early code that used buffered IO because I never bothered to check them in (shame on me). I ran into the following problems with buffered IO that mmap immediately solved: 1. The code that handles the mmap in C is much simpler, because it eliminates the need for a buffer. As far as the rest of my code is concerned, I'm simply iterating through a char* 2. I was unable to utilize a buffer of more than 10k (10,240 chars) to read data without getting a segfault. (I was allocating it using an instant array allocation at the top of my function -- I did not try using malloc.) 3. Since I'm analyzing the file byte by byte, using a buffer requires testing for the end of the buffer/end of the file after each byte, which is more slightly more complicated than simply comparing an incremented value to a limit. While I was developing, I gauged performance using a file with 20,000 lines, 424,250 words, and 3,443,296 characters (which I do still have). I did both a bulk read (returning an array of arrays of the entire file) and an iterative read (where each row is yielded to a block as an array; the block in this case joins the row with a tab.) For a file this size, there was no noticeable difference. Performance parsing this file using either read method (iterative or bulk) was as follows: 4. Using C open/close with file descriptor: approx. 10 seconds to process 5. Using mmap: just over 2 seconds to process Simple profiling gave the following results: 6. running ruby -rprofile for bulk read gave me the following: % cumulative self self total time seconds seconds calls ms/call ms/call name 71.59 3.78 3.78 1 3780.00 5200.00 Text_CSV.decode 26.89 5.20 1.42 20000 0.07 0.07 Array#clone 0.95 5.25 0.05 5 10.00 16.00 Kernel.require ... (Keep in mind that the program runs significantly slower using -rprofile, so that the time indicated above do not reflect the real, non-profiled execution time) 7. running ruby -rprofile for iterative read give me the following: % cumulative self self total time seconds seconds calls ms/call ms/call name 75.94 4.64 4.64 1 4640.00 6050.00 Text_CSV.decode 23.08 6.05 1.41 20000 0.07 0.07 Array#join 0.49 6.08 0.03 5 6.00 12.00 Kernel.require .... (Keep in mind that the program runs significantly slower using -rprofile, so that the time indicated above do not reflect the real, non-profiled execution time) Lastly, if my memory serves me correctly, then the Perl Text_CSV module (which uses stdio.h) is about the same speed as my ruby that used the unistd.h functions, and so it is much slower than my ruby routine that uses mmap. Best, Dave ------------------------------------------------------- David King Landrith (w) 617.227.4469x213 (h) 617.696.7133 One useless man is a disgrace, two are called a law firm, and three or more become a congress -- John Adams ------------------------------------------------------- public key available upon request