From: Robert Klemme Date: 2009-12-12T04:45:16+09:00 Subject: Re: Help on searching large sets of data On 12/11/2009 06:50 PM, Blarg@sogetthis.com Mr balla wrote: > I am reaching out to get some help/ideas about our current search > implementation. > > We are currently using Ferret as our search engine and its not keeping > up with our needs... we are quite needy ;). We have about 5m+ documents > that total around 8-10gb of data and indexing can take up to a half a > day which is obviously not where we want to be. > > I've been looking into Solr and Sphinx and I'm am wondering if anyone > would have any thoughts to which would be suite our needs. What are the > pros/cons of each? We would have to know what your needs are for that. So far I have understood that you have a particular volume of files you want to text index but and search. I have no idea what kinds of documents you have and what searches you want to be able to do (only words, combinations etc.). Even with your life example I'm not really seeing things clearer (might be due to my lack of knowledge about Ferret and their website being unresponsive to me). > Another search related question I have is how would one go about > classifying documents based on content? Does Solr or Sphinx have this > capability? Solr doesn't - judging from their website. > Even better question is how do I go about displaying > categories (and their counts) for a specific search term? For example if > I search for 'hat' I want a category browse like the following: > > # Accessories (30,940) > # Children (24,090) > # Crochet (19,884) > > How is this done? FYI for a live example of what I want check out: > http://www.etsy.com/search_results.php?search_type=handmade&search_query=hat > > Any help would be greatly appreciated. Thanks If you actually switch, this might be of help http://blog.xing.com/2009/07/migrating-our-search-from-ferret-to-solr/ Kind regards robert -- remember.guy do |as, often| as.you_can - without end http://blog.rubybestpractices.com/