From: Jim Freeze <jim@...>
Date: 2003-04-17T22:54:08+09:00
Subject: Re: Need IO Optimization help

On Thursday, 17 April 2003 at 22:46:04 +0900, Brian Candler wrote:
> On Thu, Apr 17, 2003 at 10:34:53PM +0900, ts wrote:
> > >>>>> "J" == Jim Freeze <jim@freeze.org> writes:
> > 
> > J> require 'mmapio'
> > J> MMapIO.new(writefile, "w") { |f|
> > 
> >    Good luck, for this
> > 
> > J>   MMapIO.new(readfile).each { |line
> > J>     # start rockin'
> > J>     f << line
> > J>   }
> > J> }
> > 
> > 
> > J> Has this been written before I start to do it myself?
> > 
> >  yes, but don't expect that it will be faster ...
> 
> I agree completely, because I've run essentially that program (but using
> Ruby's standard I/O library) through a profiler. It spends around 5% of its
> time doing I/O.
> 
> The problems are:
> - you want to process the file a *line* at a time
> - you are allocating a new String object for each line
> - you are calling 'yield' on a Ruby block for each line

For a 260MB file, ruby spent 30% of its time in io and
70% processing the lines. (In this test case we were
doing minimal processing.)

So, of that 30% (which was 80 seconds), what would a fair 
estimate be of time saved if we swapped mmap for rb_io_gets?


-- 
Jim Freeze
----------
One difference between a man and a machine is that a machine is quiet
when well oiled.