From: Duane Johnson <duane.johnson@...>
Date: 2005-11-03T02:52:24+09:00
Subject: Re: The "perfect" ORM?


On Oct 30, 2005, at 5:52 AM, Dave Burt wrote:
>> Basically the only thing missing from a YAML solution or something is
>> the ability to do sophisticated queries without storing all the  
>> objects
>> in memory at once.
>>
>> Given that I may have 100,000 objects or so, I don't want to store  
>> them
>> all in a giant array, but I *do* want to be able to find them by the
>> values of their accessors.
>
> So Marshalled objects plus indexes?
>

Nice summary, Dave.  If this is indeed what Hal is talking about, it  
seems like a very nice "fit".

 From off the top of my head, an ideal data repository has the  
following qualities:

1. Infinite storage capacity
2. Zero access time
3. Persistent / Failsafe

Current technology (i.e. a hard drive) marries persistence with  
storage capacity and unfortunately increases access time.  In-memory  
data reverses the advantages and disadvantages--it decreased access  
time, but it has a smaller storage capacity and it is no longer  
persistent.

Indexing is a netherworld.  By imposing some structure on the data  
(e.g. "The 'id' attribute will always contain an integer") we can  
store ordered information about an otherwise haphazard data web.  The  
ordering gives us the ability to predict where to look for  
information (e.g. sort the 'id' attribute numerically).  This is  
important--we only need structure where we need to predict  
something.  If we don't need to predict where to find information  
then an index is unnecessary.  The "imposed structure" of database  
tables goes away if we don't need a bird's eye view of the data.

It seems that "Marshalled objects + Indexes" gives us this happy  
middle ground--most of the time we don't need to predict where to  
find information (e.g. many array attributes) but in the cases where  
we do, we could impose that "thread" of structure (aka an index) on a  
YAML file.

Duane Johnson
(canadaduane)