From: Robert Klemme <shortcutter@...>
Date: 2006-11-16T19:00:05+09:00
Subject: Re: *Fast* way to process large files line by line

On 16.11.2006 10:51, Neil Wilson wrote:
>> Btw will using something like an mmap extension for ruby speed things
>> up for me ?.
> 
> Can't harm to try. I use it for uploading large files and it helps
> there.

I beg to differ: it can actually harm to apply an optimization measure 
without making sure that it actually solves the problem.  As long as you 
do not know what makes your program slow trying out measures is pretty 
much a waste of time.

> Unfortunately Operating systems don't always behave like you think they
> ought to when doing something like this. Get to know your profiler and
> brush up on your data processing techniques. When you hit a speed
> problem, you *have* to start and understand what the machine is doing
> with each high level instruction you give it and how they all interact
> with each other.

Exactly.  I'd start with vmstat to find out whether we are IO bound or 
CPU bound.

Regards

	robert