From: Kirk Haines <khaines@...>
Date: 2004-06-30T07:58:57+09:00
Subject: Re: [ANN] arrayfields-3.0.0

On Wed, 30 Jun 2004 06:37:56 +0900, Ara.T.Howard wrote

> and i was reading my own code thinking - huh?  so then i started doing:
> 
>    res = pgconn.exec sql
> 
>    tuples = res.result
>    fields = res.fields
> 
>    tuples =
>      tuples.collect do |tuple|
>        h = {}
>        fields.each do |field|
>          h[field] = tuple.shift
>        end
>        h
>      end
> 
> this is NOT good if tuple.size == 42000!!  finally i wrote the arrayfields
> module.  it still requires work:
> 
>    res = pgconn.exec sql
> 
>    tuples = res.result
>    fields = res.fields
> 
>    tuples.each{|t| t.fields = fields}
> 
> but now all 42000 tuples SHARE a copy of fields for doing their 
> lookups - i still have to iterate over all of em - but i don't need 
> to create any new objects and my code new reads like:
> 
>    if tuple['name'] =~ pat or tuple['age'].to_i < 42
>      ...
>    end
> 
> which is a lot nicer.

Okay, admittedly, sometimes I'm too dense for my own good, but I'm 
confused.  It looks to me like, internally, each array is using a hash to 
keep track of the fields to indices mapping, and that hash is not shared 
amongst arrays with the same set of fields.  So you end up using more memory 
using arrayfields for something like database result sets than if you just 
used hashes.

I put together a few simple little test programs to try it out, using 
arrayfields and using hashes to store similar sets of data, and in practice 
it looks like arrayfields uses almost 2x the amount of RAM for a given data 
set, as well.  Am I missing something, here?


Thanks,

Kirk Haines