From: Todd Gillespie Date: 2001-12-08T05:15:21+09:00 Subject: [ruby-talk:27849] Re: DBI and large result sets "James Britt (rubydev)" wrote: : Just to be sure I really do understand, suppose I have some code making : a database connection to some remote machine. I have an SQL query : I know will locate millions of rows. : Running : dbh.execute("SELECT * FROM MILLIONS_OF_EMPLOYEES") do |sth| : sth.fetch do |row| : p row : end : end : means the local code only has a single row in memory at any given time. That's right, if you ignore whatever caching your DB is doing, or the DB client libraries. But your query is still reading the entirety of one or many tables. Your performance is trash, even after optimizing for DBI's memory consumption. I feel fairly safe in guessing that your application does not print several million rows of unordered data directly into the user interface, so I'm assuming you do some filtering and operations on your millions of rows in-application. You would be much better off doing this with a more complex query or a stored procedure.