From: Dave Fayram <dfayram@...>
Date: 2006-02-10T03:38:24+09:00
Subject: Re: Using Classifier::LSI

>Ok, so it took just over 7 hours to build the index of 3000 items.

Depending on the size of your inputs, it can take longer.
Classifier::LSI has a very specific sweet spot of data volume in its
current incarnation. It's not currently suited for very large loads. I
am currently working on a version that will work with the MPI so that
clusters can attack the problem. LSI just isn't useful for large
datasets without _very_ hefty hardware.

But if your matrix is only 3000x4000-ish, it shouldn't take that long.
Make sure that your classifier is binding to gsl, and if you need to,
email me and we can pursue it further. I only experience a breakdown in
speed when my unique-words list is in the 15k+ range, with more than
1500 items.

Also, Remember that LSI wants to work with the *semantics*. A list of
product titles isn't going to bring very much semantic data into the
system. The way I originally envisioned using LSI was to create
summaries of documents, and working on sets of document abstracts to
rapidly provide relevant ACM papers. :)