From: "Mauricio Fernández" <batsman.geo@...>
Date: 2003-04-18T00:21:27+09:00
Subject: Re: Need IO Optimization help

On Thu, Apr 17, 2003 at 11:56:15PM +0900, Jim Freeze wrote:
>  
> > Did your program actually spend 30% of its time in fread()? Or was it 30% of
> > its time in rb_io_gets? It's an important distinction.
> 
> Of the original 30%, 30% of THAT time was system time (which I assume

I believe that 30% is time spent in kernel mode, doing real IO.

> to be fread() and not rb_io_gets). So, if mmap() was infinitely fast,
> I could save 9% overall. 


> Or, since we combine continued lines, it looks like we could only save
> a fraction of 20% overall if we rewrote rb_io_gets to not create string objects for
> every line, but for the 'final' combined line.
> My thought is that if I could save another 10% that would be good,
> but going below rb_io_gets is too expensive for the savings considering
> that we are already 15% faster than perl.
> 
> > 
> > As a guess, I'd say your savings would be minimal. Should you decide to mmap
> > the whole 260MB into your address space at once you may actually get worse
> > overall performance, unless you have that amount of free RAM available.
> > 
> 
> I agree. We can't do that anyway, because the 260MB file is just a test
> file. Most are around 950MB.

I think you can actually use mmap: just map different parts of the
file each time, at a different offset. You might save some kerner/user
switches, but it won't make that much of a difference.

-- 
 _           _                             
| |__   __ _| |_ ___ _ __ ___   __ _ _ __  
| '_ \ / _` | __/ __| '_ ` _ \ / _` | '_ \ 
| |_) | (_| | |_\__ \ | | | | | (_| | | | |
|_.__/ \__,_|\__|___/_| |_| |_|\__,_|_| |_|
	Running Debian GNU/Linux Sid (unstable)
batsman dot geo at yahoo dot com

How should I know if it works?  That's what beta testers are for.  I
only coded it.
	-- Attributed to Linus Torvalds, somewhere in a posting