From: Chris Kottom <chris@...>
Date: 2011-05-27T03:32:42+09:00
Subject: Doing LSI at scale in Ruby

--20cf305e2533cc028104a4320bbd
Content-Type: text/plain; charset=UTF-8

Hi all,

I'm looking to find out whether anyone is doing latent semantic indexing
(LSI) in Ruby at any kind of web scale, and if so, what tools and techniques
you're using?

Just for context, I've been working on this problem for a few days now.
 I've tried the Classifier gem via "gem install" and compiled from source
and at least two other forks.  I've tried compiling various versions of the
GSL library, most of which would not allow the gsl gem to compile, and it
seems that in the combinations where I can actually get the full set of
libraries to install, I receive an error like the following:

/home/ck1/.rvm/gems/ruby-1.9.2-p180@classifier_test/gems/kitop-classifier-1.4.4/lib/classifier/lsi.rb:316:in
`SV_decomp': Ruby/GSL error code 24, svd of MxN matrix, M<N, is not
implemented (file svd.c, line 61), the requested feature is not (yet)
implemented (GSL::ERROR::EUNIMPL)
        from /home/ck1/.rvm/gems/ruby-1.9.2-p180@classifier_test/gems/kitop-classifier-1.4.4/lib/classifier/lsi.rb:316:in
`build_reduced_matrix'
        from /home/ck1/.rvm/gems/ruby-1.9.2-p180@classifier_test/gems/kitop-classifier-1.4.4/lib/classifier/lsi.rb:128:in
`build_index'
        from /home/ck1/.rvm/gems/ruby-1.9.2-p180@classifier_test/gems/kitop-classifier-1.4.4/lib/classifier/lsi.rb:66:in
`add_item'
        from lsi_test.rb:18:in `block in <main>'
        from lsi_test.rb:18:in `each'
        from lsi_test.rb:18:in `<main>'

This particular stack trace was when running with a fork of Classifier, but
the result is essentially the same with the original gem with the exception
of the line numbers, and it looks as though the error is unrelated to
Classifier but rather the gsl gem or the underlying GSL library.

Any help or shared experiences will be appreciated.  Thanks in advance.

--20cf305e2533cc028104a4320bbd--