Full Text Searching with Solr and Sunspot(collectiveidea.com)
collectiveidea.com
Full Text Searching with Solr and Sunspot
http://collectiveidea.com/blog/archives/2011/03/08/full-text-searching-with-solr-and-sunspot/
2 comments
Good intro. I've been using sunspot for a year on a ruby webapp (merb, I found the sunspot lib nicely framework agnostic). It took a few hours to setup and get my content indexed. Its been running in production for a year with zero maintenance.
Thanks jhancock - I'm planning on writing a post talking about sunspot without sunspot_rails. I tried to make this post concise. It's cool to hear you have already had success with it.
Solr/lucene (they're on a joint release schedule now) give you more config options than any other fulltext engine. Some alternatives are sphinx (blazing fast indexing), IndexTank, and Riak Search (I haven't tried the last 2).
If anyone has any good resources on Riak search, please share them. I keep hearing good things about Riak and haven't dived in yet.
Check out Sphinx and the amazing ThinkingSphinx gem. Having used both extensively, I greatly prefer Sphinx over Solr.
http://sphinxsearch.com/
http://freelancing-god.github.com/ts/en/
CollectiveIdea also maintains a good fork of delayed_job for those who are interested in that.
http://sphinxsearch.com/
http://freelancing-god.github.com/ts/en/
CollectiveIdea also maintains a good fork of delayed_job for those who are interested in that.
Thanks getsat for the shout out. Our guys have put in a lot of work adding features and keeping delayed_job up to snuff.
I do have to disagree with you on ThinkingSphinx. I have used both, and I gel much better with Solr. Don't get me wrong, the guy(s) who did ThinkingSphinx have worked hard. I just don't like the moving pieces of ThinkingSphinx: continuous rake tasks, the fact that it sits directly on top of the database, extra re-deployment steps when using capistrano. I prefer having the search server sitting apart.
I do have to disagree with you on ThinkingSphinx. I have used both, and I gel much better with Solr. Don't get me wrong, the guy(s) who did ThinkingSphinx have worked hard. I just don't like the moving pieces of ThinkingSphinx: continuous rake tasks, the fact that it sits directly on top of the database, extra re-deployment steps when using capistrano. I prefer having the search server sitting apart.
Sphinx does have the advantage that it runs fine on a little 256 Meg VPS, where you're not going to shoehorn in Tomcat or Jetty
This and Sphinx's insanely fast indexing speed are two of the main reasons I like it so much. Java is a huge dependency in my opinion.
Different strokes for different folks. :)
Different strokes for different folks. :)
Yeah, i would say insanely fast indexing speed is a winner at 3 stages of
a) developing the 2 or 3 indexes you need for good search results b) running test suites c) production
That said, in the past, I haven't been able to transfer the index configuration from sphinx to SOLR indexes and get 98% matches between the 2 engines. Sphinx (thinking sphinx in rails) used to do funny stuff in the pre-1.0, like if you put in 2 search terms you would get X results, if you put in those 2 terms plus another, you would get more results. I think they've fixed most of that in 1.0
a) developing the 2 or 3 indexes you need for good search results b) running test suites c) production
That said, in the past, I haven't been able to transfer the index configuration from sphinx to SOLR indexes and get 98% matches between the 2 engines. Sphinx (thinking sphinx in rails) used to do funny stuff in the pre-1.0, like if you put in 2 search terms you would get X results, if you put in those 2 terms plus another, you would get more results. I think they've fixed most of that in 1.0
First of all, Sphinx is fantastic and a majority of people will find it suitable for full-text search, but there are some weaknesses that might make Solr (Lucene, really) more powerful and appropriate for full-text:
1.'did-you-mean' support
2. facets
3. replication
4. spell checking
5. auto-complete is easier to implement with Solr
6. Sphinx doesn't return whole documents, only the doc id
7. Sphinx doc ids must be integers
1.'did-you-mean' support
2. facets
3. replication
4. spell checking
5. auto-complete is easier to implement with Solr
6. Sphinx doesn't return whole documents, only the doc id
7. Sphinx doc ids must be integers
If you have a really huge dataset go away from sphinx, it takes forever to index and doesn't offer incremental updates, as well it crashes regularly
No, there is no incremental mode to the indexer, but that doesn't mean it can't be done.
Create a counter table to keep track of what records you've indexed and cron frequent indexer runs using the latest records as offsets.
If you have documents that frequently change, maintain your full index less often and instead merge the two on a more regular basis.
Sphinx's indexer speed is one of its advantages, but it's largely dependent on the efficiency of your SQL and underlying MySQL indices. Perhaps something else is indirectly influencing your indexing performance.
Create a counter table to keep track of what records you've indexed and cron frequent indexer runs using the latest records as offsets.
If you have documents that frequently change, maintain your full index less often and instead merge the two on a more regular basis.
Sphinx's indexer speed is one of its advantages, but it's largely dependent on the efficiency of your SQL and underlying MySQL indices. Perhaps something else is indirectly influencing your indexing performance.
What size dataset are you talking about? I've never seen Sphinx crash even when indexing hundreds of gigs of documents. Are there any relevant bug reports?
Yeah, we use a large dataset (so does craigslist, etc). I've not seen crashes. I'd be curious to hear more about what you ran into.
The indexing speed is so very fast-- I can't say enough good things about sphinx.
The indexing speed is so very fast-- I can't say enough good things about sphinx.
care to share the reasons of this preference?
I use both; I prefer sunspot if I need to index arbitrary ruby methods results (eg: methods that result in multiple queries/data clean-up) or if I work with MongoDB. I prefer sphinx when I need a more lightweight setup on RAM, or need an easy geosearch, or work with mysql.
EDIT:
- sunspot allows for per-request weighting in the search (see http://blog.logeek.fr/2011/3/4/hackerbooks-books-stackoverfl... for an example), not sure if it's possible with ts
EDIT:
- sunspot allows for per-request weighting in the search (see http://blog.logeek.fr/2011/3/4/hackerbooks-books-stackoverfl... for an example), not sure if it's possible with ts
That flexible weighting looks nice! Thanks for the link.
Yeah, if I ever use a NoSQL datastore on a project I'll likely end up using Solr. Faster than writing a new Sphinx interface for it.
Yeah, if I ever use a NoSQL datastore on a project I'll likely end up using Solr. Faster than writing a new Sphinx interface for it.
Easier to setup, doesn't depend on Java, insanely fast indexing, delta indexes. Basically, it's just a fast little search server that handles everything I've thrown at it.
That, and I had a terrible experience using acts_as_solr back in the day...
That, and I had a terrible experience using acts_as_solr back in the day...