From: Jano Svitok <jan.svitok@...>
Date: 2008-04-08T16:43:36+09:00
Subject: Re: heavy loop functions slow

On Tue, Apr 8, 2008 at 7:17 AM, Michael Linfield <globyy3000@hotmail.com> wrote:
> Alright so I was playing with my large amounts of data and ran into yet
>  another problem with shoving it into a loop that requires a substantial
>  amount of memory.
>
>
>
>  dataArray = []
>  output = arrayOut.to_s.chop!.split(",")

set arrayOut to nil if you don't need it any more.

>  output1 = output[0..356130]
>  output2 = output[356131..712260]
>  output3 = output[712261..1068390]
>  output4 = output[1068391..1424521]

You dont need output here, set it to nil to allow for garbage collection

>  count = 0
>     output1.each do |out|
>       out = out.to_i
>       push = hashRange[out]
>       dataArray << push
>       count+=1
>       puts "#{push} - #{count}"     #Testing purposes
>     end

1. you can convert the output to numbers in one pass, though use
benchmark to see the actual gain:

output = arrayOut.to_s.chop!.split(",").map {|out| out.to_i }

2. if you are looking for numbers only, you can do something like

output = []
arrayOut.to_s.chop!.scan(/\d+/) {|out| output << out.to_i }
(you can count the items, and switch to output2 when output1 has
enough, thus 1. creating smaller arrays, 2. doing two things in one
step.)

3. even in this case, you still have both the original arrayOut and
the long string (.to_s) in memory.
It might be faster, if you could iterate through the array without
creating the intermediate string. The question is 1. will it help? 2.
Is it worth it?