From: Kevin Hopkins Date: 2010-04-24T03:23:05+09:00 Subject: Re: When to use a db? --001636b14ca34cc38f0484eb68ec Content-Type: text/plain; charset=ISO-8859-1 I recently had a project here at work where I had to take close to 120k products and format them from a csv into a set of necessary attribute changes and then wrote them to a .tab file. I had experimented with a few different methodologies for manipulating the data and iterating over with it and by far the fastest way was just to create a db, pump the data into it via the csv, and read from the model and write to a csv. I was working with segments initially that were only about 1500 records in size and they performed faster in ram than through a model. It seems once I broke the 10k mark of records, performance increased about 50% by switching to the model base. I think codewise, looking up whether "X exists" would probably be easier using the model than trying to find it in the csv as well as faster with the db indexes but it all depends on how many records you have. If you want to reply with how many records/attributes you have or with just maybe the size on the filesystem that the csv is, I can give a better appraisal but it really is up to you. If you're already doing it by storing it in RAM and its working fine, done change it. But if you want to squeek some more time out of the project, the database may be worth a shot. Hope this helps, I know its a long winded answer... --Kevin On Wed, Apr 21, 2010 at 12:11 PM, Moya Pilot wrote: > Dumb question but I am reading in a constant stream of lines via > fastercsv and pushing it into an array. Each line is either a new > object or will update some information in an existing object. 98% of > all lines will be an update to an object. I will also be doing a lot of > lookups on the objects to see if X exists somewhere. I don't have too > much experience with this stuff but I have read and been told by a > couple people that if I am just iterating repeatedly, updating and doing > a bunch of lookups on objects then perhaps a db would be more > efficient/faster. > > I know there isn't a sure fire threshold that states when you are > dealing with X amount of data you should use a database but I always > assumed that if I am using an array it would be faster since it is in > memory. I am not opposed to using something minimal like SQLite I just > wasn't sure if things would be easier/faster/more efficient to use a db. > Object Persistence is a plus but not a necessity for me, if my script > dies and the array is lost it isn't a huge deal. > > Will using a SQLite DB be faster or easier or more efficient for me than > just doing all my iterations thru an array? > > Thanks > -- > Posted via http://www.ruby-forum.com/. > > -- Kevin Hopkins, MCP MCTS (c) 540.331.1772 (e) kevin@wearefound.com (w) www.wearefound.com --001636b14ca34cc38f0484eb68ec--