From: Jim Freeze <jim@...>
Date: 2003-04-17T23:56:15+09:00
Subject: Re: Need IO Optimization help

On Thursday, 17 April 2003 at 23:14:13 +0900, Brian Candler wrote:
> On Thu, Apr 17, 2003 at 10:54:08PM +0900, Jim Freeze wrote:
> > > The problems are:
> > > - you want to process the file a *line* at a time
> > > - you are allocating a new String object for each line
> > > - you are calling 'yield' on a Ruby block for each line
> > 
> > For a 260MB file, ruby spent 30% of its time in io and
> > 70% processing the lines. (In this test case we were
> > doing minimal processing.)
> > 
> > So, of that 30% (which was 80 seconds), what would a fair 
> > estimate be of time saved if we swapped mmap for rb_io_gets?
> 
> I don't think that's a meaningful question, because mmap() doesn't do the
> same thing as rb_io_gets.
> 
> I think what you are asking is about rewriting rb_io_gets to do the same
> things it currently does (i.e. divide the input into lines, allocate a
> string for each one), but using mmap() instead of fread() to access the
> file.
 
 Yes.
 
> Did your program actually spend 30% of its time in fread()? Or was it 30% of
> its time in rb_io_gets? It's an important distinction.

Of the original 30%, 30% of THAT time was system time (which I assume
to be fread() and not rb_io_gets). So, if mmap() was infinitely fast,
I could save 9% overall. 
Or, since we combine continued lines, it looks like we could only save
a fraction of 20% overall if we rewrote rb_io_gets to not create string objects for
every line, but for the 'final' combined line.
My thought is that if I could save another 10% that would be good,
but going below rb_io_gets is too expensive for the savings considering
that we are already 15% faster than perl.

> 
> As a guess, I'd say your savings would be minimal. Should you decide to mmap
> the whole 260MB into your address space at once you may actually get worse
> overall performance, unless you have that amount of free RAM available.
> 

I agree. We can't do that anyway, because the 260MB file is just a test
file. Most are around 950MB.


-- 
Jim Freeze
----------
We really don't have any enemies.  It's just that some of our best
friends are trying to kill us.