From: Philip Rhoades <phil@...>
Date: 2008-06-27T23:47:55+09:00
Subject: Re: Performance improvement possible?

Ellie,


Eleanor McHugh wrote:
> On 27 Jun 2008, at 07:55, Philip Rhoades wrote:
>> Eleanor McHugh wrote:
>>> I'm surprised at the behaviour you're seeing so I ran a quick 
>>> benchmark on what I believe to be an equivalent array handling 
>>> problem with my laptop (OS X, Ruby 1.8.6-p111, Core 2 Duo 2GHz, 2GB 
>>> RAM):
>>> require 'benchmark'
>>> include Benchmark
>>> bm(6) do |y|
>>>   y.report("appending") { x = []; (1..20000000).each { |i| x << i } }
>>>   y.report("nested creation") { x = Array.new(1000) { Array.new(1000) 
>>> { Array.new(20, 0) } } }
>>>   y.report("unrolling") { x.flatten.flatten.length }
>>> end
>>>                        user     system      total        real
>>> appending        7.350000   0.100000   7.450000 (  7.685458)
>>> nested creation    0.850000   0.000000   0.850000 (  0.851341)
>>> unrolling        15.240000   0.280000  15.520000 ( 20.927189)
>>> which in each case is manipulating an array of 20,000,000 elements. 
>>> This would be equivalent to processing 32000 files where each 
>>> contained 625 occurrences of the 'k=' tag.
>>
>> I haven't used BM yet but I can't see how doing a benchmark on an 
>> array helps?  When I had an array it was slow, when I ditched the 
>> array it was fast?
> 
> Your experience runs counter to all of my experience, both in terms of 
> several years of Ruby hacking and more generally in working on 
> performance-optimisation for large data manipulation problems. Therefore 
> being of a curious disposition I'd love to figure out why :)


I don't think there is any great mystery here - in the first version I 
thought it would be efficient to read the entire text file into an array 
  and then process the array, line by line.  In the second case, I 
eliminated the array and processed the data directly after reading it 
(the same as I was doing in the first case) and relied on the Linux 
system buffers/cache to be efficient about reading the data from disk. 
I think the processing time saved (~50%) is just a function of not 
putting data into an unnecessary structure and not manipulating the 
structure?

Regards,

Phil.
-- 
Philip Rhoades

Pricom Pty Limited  (ACN 003 252 275  ABN 91 003 252 275)
GPO Box 3411
Sydney NSW	2001
Australia
E-mail:  phil@pricom.com.au