From: Christian Madsen Date: 2006-10-28T17:00:13+09:00 Subject: Re: Efficient parsing of large Excel documents in Ruby You should definitely use win32ole. This way, you can cut down memory usage to the minimal needs by only storing the current row. I've been using it for storing 11000 rows by 10 columns with great success. However, there seems to be some kind of round trip delay in the OLE communication: writing 11000 rows takes about 90 seconds, but that should still be OK for you. I overcame this by simply storing an 11000 by 10 array, which took ~1 second. This then required the memory usage of an 11000 by 10 array, but not 200MB... See this link: http://wiki.rubygarden.org/Ruby/page/show/ScriptingExcel So in the end, you should use win32ole and probably make an Excel wrapper class for instantiating/closing Excel, appending rows etc. -Christian Wes Gamble skrev: > All, > > I am currently using the parseexcel gem to parse an Excel file so that I > can save it's data into a database (in a Rails app.). > > When I run it against an Excel file with ~42000 rows and 11 columns (the > spreadsheet is about 10MB in size), it takes the better part of 10 > minutes to parse the file and uses about 200 MB of memory. I'm not > actually sure if the parsing completes successfully - the app. stops > right after the parsing appears to finish. > > It would seem that I have two issues - memory usage and performance. > > Is anyone aware of a much faster way to parse large Excel files? > > Is anyone aware of a "windowing" scheme that uses a set amount of memory > to transfer portions of the data to a client requesting the data so as > to keep the memory required at a fixed level? > > Should I look into using the Win32OLE libraries with the hope that it > would be more efficient? > > I am transferring this data into a SQL Server database, so perhaps I > could look into some M$-native method of moving this data. > > Any thoughts or advice is appreciated. > > Thanks, > Wes Gamble > > -- > Posted via http://www.ruby-forum.com/.