From: David Rush Date: 2007-07-28T22:45:03+09:00 Subject: Re: Usage patterns and trusted libraries On Jul 27, 8:13 am, "Robert Klemme" wrote: > 2007/7/27, David Rush : > > The process takes over 12 hours to run using ActiveRecord to provide > > my DB access. For 5500 records. > > > So eventually I rewrote the program using a different language (which > > I don't mention to avoid starting a flame-war - I could have used Ruby > > on top of the MySQL interfaces) with a cache strategy that is better > > suited to the DB access pattern. > > > The new process takes 55 *seconds*. > > Makes me wonder: why did you choose a different language? Because I found the Ruby AR code to be so heavily metaprogrammed, so inbred, and so weakly documented (internals, not API) that after three days of crawling around inside the code I figured that it would be faster to hack together a weak system that was based on the queries, rather than the relatively nice model I had to work with for the UI. And because in other languages I know the assumptions made by the interpreters, virtual machines and compilers in much greater detail and can therefore accurately code for CPU-efficiency. > As you said > yourself, you could have implemented the same strategy in Ruby as > well. Also, it seems for 5500 records you do not need a caching > strategy - you could just slurp in all the stuff into mem, do your > transformations and write it back. It seems with this approach even > AR would have provided sufficient performance, wouldn't it? One would have thought. I did slurp all 5500 (BTW, this is and intermediate load of a total data set of 25,000 - I have another dataset of 20,000 waiting in the wings, and a deal I'm doing for another million or so :). Those 5500 required another 26000+ records of various other model types in order to get them properly embedded in the model - which is still not a big deal. However, slurping the *entire* DB of over 250,000 records seemed like a potentially bad idea, given that I don't have a good feel for the memory overheads in Ruby yet. I had a lot of difficulty figuring out where to hook the AR find mechanisms to avoid multiple searches for the same records. Looking back, I think I could do it now, but that is also because I have dropped my expectation of what AR (or various net-available plugins) will do and would just write my own access layer on top of AR. Don't you just love the perpetual recurrence of the glueware-on- glueware antipattern? david rush -- http://cyber-rush.org/drr -- a very messy construction^Wweb site