From: Peter Hicks Date: 2011-08-31T04:17:16+09:00 Subject: Importing flat-file data in to a database with Ruby All, I'm up against a performance issue with some code I have - it's on GitHub at https://github.com/poggs/tsdbexplorer/blob/master/lib/tsdbexplorer/cif.rb if anyone wants to take a look. The code takes a fixed 80-column wide input file, reads the first two bytes of each line and hands off further processing to an initialize function in a Class. An object is returned and pushed on to an Array. When ~1000 of these records have been processed, they are INSERTed in to a database using activerecord-import. Without calling the import method on an ActiveRecord object, I can process a 450Mb source file in 6-8 minutes. When writing to the database, this easily takes in excess of an hour. Can anyone help me work out where I'm going wrong? Both MySQL and PostgreSQL show this 'slowness', so I'm looking at code first. Kind regards, Peter -- Peter Hicks