From: Philip Rhoades <phil@...>
Date: 2008-06-27T05:29:17+09:00
Subject: Re: Performance improvement possible? - update

Eleanor,


Philip Rhoades wrote:
> Eleanor,
> 
> 
> Eleanor McHugh wrote:
>> On 26 Jun 2008, at 04:24, Rob Biedenharn wrote:
>>> On Jun 25, 2008, at 8:44 PM, Philip Rhoades wrote:
>>>> Chuck,
>>>> Chuck Remes wrote:
>>>>> On Jun 25, 2008, at 5:13 PM, Philip Rhoades wrote:
>>>>>>
>>>>>> The change to a single Ruby script simplified the previous setup 
>>>>>> quite markedly but I'm not sure that much more can be done with 
>>>>>> the relatively small script now.  I logged into "ruby Talk" on 
>>>>>> http://codeforpeople.com/ but I couldn't see anywhere to post the 
>>>>>> code.  I can mail the files to anyone who is interested in having 
>>>>>> a look.
>>>>> Phil,
>>>>> use one of the pastie sites to paste up your code for us to see.
>>>>> http://pastie.org
>>>>> http://rafb.net
>>>>
>>>> Done - http://pastie.org/222306
>>>>
>>>> I am also rerunning with v1.9 and now that the code is all in one 
>>>> program it seems to be faster (but still about 3x slower than the 
>>>> equivalent shell scripts and C/C++ program).
>>>>
>>> OK, even knowing that it started as shell+C, it needs help ;-)
>>>
>>>  http://pastie.org/222375
>>>
>>> Except for noticing that the Dir.chdir is still commented out, I 
>>> think that this is equivalent to your original.  Of course, all that 
>>> really means it that I think it produces the same output.  I don't 
>>> have your input files to check (and I'm not really sure I have a clue 
>>> what it is really doing either).
>>>
>>> If you can directly try this new set, I'd be curious as to the 
>>> relative speed.  A few important notes:
>>> * the $darr was never referenced so why bother to initialize it
>>> * the multiple loops in stats06 have been collapsed
>>> * the various seed* loops now use strings directly
>>> * the parameters to statz32000 are strings rather that do somewhat 
>>> pointless conversions (and sample_sz of 999 would 'overflow' the %02d 
>>> into "999" anyway).  Some potential values won't work as literals 
>>> (try 08 as an example, or 012) since the leading 0 on a literal means 
>>> octal.
>>> * the more common boolean values are true (rather than TRUE) and 
>>> false and parentheses are not generally needed around the conditional 
>>> expression of an 'if'
>>
>>
>>
>> A further modified version of statz which should changes the way IO is 
>> handled:
>>
>>     http://pastie.org/222765
>>
>> Let me know if this makes any difference.
> 
> 
> A few things:
> 
> - you left a line in the loop:
> 
>     File.open( output_filename, 'w' ) do |fout|
> 
> which should be deleted
> 
> - I originally used:
> 
>     stats = []                      lines = 
> File.readlines(input_filename, 'r')
> 
> but found that reading the whole file (8871 lines) and then processing 
> the array was inefficient so I got rid of the array
> 
> - using:
> 
>     stats << stats06
> 
> and the file writing output of:
> 
>         File.open(output_filename, "a") do |file|
>         file.sync = false
>         file.puts *stats
>         file.fsync
>     end
> 
> looks interesting - why should that be faster?


I tried that mod - it doesn't make any difference to the speed but it 
thrashes the disk because of all the syncing.  The output files are very 
small (~110 bytes) so it is probably not very efficient on 32,000 small 
files but it might be good for big files?

Thanks,

Phil.
-- 
Philip Rhoades

Pricom Pty Limited  (ACN 003 252 275  ABN 91 003 252 275)
GPO Box 3411
Sydney NSW	2001
Australia
E-mail:  phil@pricom.com.au