From: Philip Rhoades <phil@...>
Date: 2011-03-18T08:28:27+09:00
Subject: Re: Inserting hash value slows down as table gets larger

Brian,


On 2011-03-18 04:04, Brian Candler wrote:
> Dave Baldwin wrote in post #987924:
>> set to
>>         Array.new(2){ Array.new(1){ Array.new( 20, rand(1) ) } }
>>
>> I get 3.75 seconds
>>
>> set to
>>         # h1[ "#{a}.#{b}.#{c}.#{d}" ] = Array.new(2){ Array.new(1){
>> Array.new( 20, rand(1) ) } }
>>
>> I get 0.47 seconds
>
> Try these too:
>
>      h1[ "#{a}.#{b}.#{c}.#{d}".freeze ] = 3
>
>      h1[ "#{a}.#{b}.#{c}.#{d}".freeze ] = Array.new etc
>
> It's a foible of Ruby that if you use a String as a hash key, it is
> dup'd and frozen - unless it was frozen already, which is what the above
> does.


Adding "freeze" improves the situation by about a second on my machine:

	18.3 -> 17.3


> The other thing to beware of is YAML - try using Marshal instead as a
> comparison, because YAML might not be very efficient when dealing with
> very large data structures. I think you said you needed a human-editable
> form, but this would rule out YAML as the source of the problem.


Yes that is very slow too but that is another issue - I can live with 
that for the convenience of having human readable data . .


> If it is, then you can look at alternative YAML libraries, or at JSON.


The other thought I had was putting the data into a sqlite3 db - I will 
try it and see what happens but I don't imagine it would be faster than 
a memory based hash table?

Thanks,

Phil.
-- 
Philip Rhoades

GPO Box 3411
Sydney NSW	2001
Australia
E-mail:  phil@pricom.com.au