Show HN: TF-IDF search engine in 30 lines of Scala(github.com)
github.com
Show HN: TF-IDF search engine in 30 lines of Scala
https://github.com/felipehummel/TinySearchEngine
8 comments
Cannot edit anymore, but here is an example I had written some time ago in Python. http://www.wausita.com/2010/08/build-vector-space-search-eng...
A bit opaque, but cute. I've got a Redis + Python version (keeps the index in Redis) that I talked about last year: http://dr-josiah.blogspot.com/2010/07/building-search-engine...
As someone who works on search and is a scala fan, I'm super impressed. Next obvious question: how many lines would it take to add some sort of stemming?
Maybe in a couple of lines we could do a very naive plural removal. I've seen code for stemming, they have tons of different conditions and possibillities, I guess it would be difficult to condense that into few lines.
i don't think tf-idf relevant for tiny in-memory index >__>
The index does not have to be tiny, the source code is. I tested with a 500k documents dataset.
Out of curiosity how long did that take to index?
Actually the below implementation is usually the first non trivial thing I try to implement in any language I am learning.
EDIT - Was going to fork this on GitHub and make a quick Python port but corporate firewall got in the way.