From: Cameron McBride Date: 2006-02-08T22:14:16+09:00 Subject: Re: Using Classifier::LSI Hey Chris, On 2/8/06, chrisjroos@gmail.com wrote: > The default behaviour is to index items as they are inserted (at least > it would certainly appear that's what's happening based on simple > benchmarking). This causes the time to index new items to rise > exponentially. right. the index building takes time and there is a lot of double work going on. > After some experimentation, I found that turning off > the automatic indexing and then manually building once all items were > added was far quicker (17 secs to add 100 items with auto indexing, 0.6 > secs to add 100 items with manual index building). Great. This is exactly why there is an :auto_rebuild option, so you can make this more effecient (like you discovered). > I'm really just wondering why the default behaviour seems to be the > most inefficient and whether I'm missing anything obvious using the > manual build method? I believe the default behavior is directed toward simple usage to help give people a feel for what the library does. Given the mathematics underneath the index building, when you have 3000 items the index building is slower. (I believe the docs say around 500 is rule of thumb for speedy return). I don't think you're missing anything, here. Just use :auto_rebuild=>false and build it manually. I was actually advocating a lazy evaluation as the default (so the index wouldn't be built until you needed it, which would work rather effeciently for both simple usage and your case). The argument against this was to prevent users from thinking the searching was slow. Once the index is built, everything else flies. LSI just really is math intensive, which is why there is a C-based backend for it. Just make sure you're using something other than the pure-ruby version if speed becomes an issue. Let us know if anything else is unclear.