From: Ezra Zygmuntowicz <ezmobius@...>
Date: 2007-02-09T14:32:40+09:00
Subject: Re: General Approach to Data Validation

Hi~

On Feb 8, 2007, at 11:46 AM, Gary Wright wrote:

>
> On Feb 8, 2007, at 11:51 AM, Drew Olson wrote:
>
>> Pit Capitain wrote:
>>> The other way would be to read the records one after the other  
>>> (using
>>> cursors in db terminology) in an appropriate sort order. If  
>>> ActiveRecord
>>> allows you to do this in parallel with both tables, you could  
>>> compare
>>> the tables like this:
>>
>> This is the exact approach I originally wanted to take. I would be  
>> very
>> interested to know if ActiveRecord supports this type of operation. I
>> have yet to see a way to do it from what I've read of the  
>> documentation.
>
> A while back (last year?) I asked why ActiveRecord didn't support
> 'lazy' iteration over query results.  AR slurps up the entire
> result set into memory and then wraps objects (i.e. instances of
> ActiveRecord::Base or subclasses) around each row.
>
> This approach obviously has problems if you have large data sets.
>
> Perhaps this has changed since I last posed the question?
>
>
> Gary Wright


	ActiveRecord will start to consume a huge amount of memory and cpu  
if you fetch more then a thousand or so records at a time. This is  
because it does load all results into memory at once and then wraps  
each row in a AR object which is expensive with the amount of records  
being talked about in this thread. There is an AR plugin somewhere  
called paginating find that works sort of like a cursor but doesn't  
use a cursor, it does limits and offsets.

	But I do think there is an ActiveRecordExtensions[1] project that  
fixes some of these issues as well.

Cheers-
-- Ezra Zygmuntowicz 
-- Lead Rails Evangelist
-- ez@engineyard.com
-- Engine Yard, Serious Rails Hosting
-- (866) 518-YARD (9273)

[1]  http://rubyforge.org/projects/arext/