From: Chris Kottom Date: 2011-05-28T03:49:36+09:00 Subject: Re: Doing LSI at scale in Ruby --20cf305b0e101532b304a4466629 Content-Type: text/plain; charset=UTF-8 So for what it's worth, the particular issue I was running into was not caused by Classifier at all but rather by the test data I was using. The application this is for is still under development, so the texts that I'm indexing are multiple-paragraph blocks being generated using Faker::Lorem. The problem here is that this library has a limited vocabulary of less than 200 words, and Classifier::LSI requires that the number of unique words being indexed across all texts must be greater than or equal to the number of text instances. (It seems like it was also filtering out some number of words -- probably one- and two-character words which might be considered stop words.) So as soon as the number of records indexed exceeded the number of unique words, the underlying library (GNU GSL) propagated an exception. I've now tested this against a set of strings utilizing a richer vocabulary, and even though indexing slows down exponentially with greater numbers of records, it completes successfully. Hope this description helps someone else out. On Fri, May 27, 2011 at 10:42 AM, Chris Kottom wrote: > Thanks, Ryan. I will do this too, was just looking to see what the current > de facto standard method for this is. Digging a little deeper on both > GitHub and RubyForge, it seems that the gem has been pretty much dormant > for > several years, so I'm looking to see whether people have moved on to > another > fork or another lib. Will post any findings. > > Thanks, Clifford, for the tip. It's not exactly what I need for this > particular part of the application, as I'm using the Classifier LSI feature > to index documents and detect similar records, but it might be worth > investigating as a replacement for Sphinx in other places in this app and > others I'm working on. > > On Fri, May 27, 2011 at 3:22 AM, Clifford Heath > wrote: > > > On 05/27/11 04:32, Chris Kottom wrote: > > > >> I'm looking to find out whether anyone is doing latent semantic indexing > >> (LSI) in Ruby at any kind of web scale, and if so, what tools and > >> techniques > >> you're using? > >> > > > > The author of Picky presented it last > > night > > at the Melbourne Ruby group. Not sure if it's interesting to you, but it > > looks > > like a different kind of search engine to Sphinx, etc. > > > > Clifford Heath. > > > > > --20cf305b0e101532b304a4466629--