Many years ago, when I was building Lunr, it was initially based on IndexedDB, but it was _much_ slower and the datasets I had in mind easily fit in memory so I didn’t pursue it. No idea if that has changed since though.
Lunr.js [1] used to use a trie data structure and memory usage was a concern. The problem is that although the data structure compresses prefixes, it does nothing for suffixes. Depending on the corpus this can lead to a lot of duplication.
Switching to a trie-like structure that compresses prefixes and suffixes can lead to significant savings. Building the structure can be a bit more burdensome, so there is a trade off there. There is a paper describing the approach [2] and, if you're interested, my JavaScript implementation [3][4].
The last time this article was on HN I took a look at adding Okapi BM25 to lunr, from what I remember the changes don't seem to huge, its just a matter of getting the time to sit down and implement it!
That is almost exactly what lunr is doing. It tokenises the input text, stems the tokens and filters out any stop words. The index it can be searched, the order is not relevant, a prefix search is currently used so that you can find documents containing terms without having to type the whole term exactly. The matching documents are also scored as to how relevant they are to the search term.
In the future I want to add even more powerful querying, restricting search to specific fields, taking into account the distance between terms, and adding faceted search to reduce the total documents being searched over.
One of the original goals of the project was specifically to provide a better alternative to just using the browsers built in find-in-page functionality
Since the library can be run outside of the browser (using node.js for example) the index could be generated server side, and then just passed to the client. I hadn't considered this before but it might be worth looking at.
The example (http://lunrjs.com/example/) indexes 100 stackoverflow questions, some of which are relatively long.
If indexing performance starts to become an issue the whole search index can be moved into a web-worker, which prevents indexing from blocking the rest of the page.
I have a similar library here http://bit.ly/fTj2Ls that uses pushState to provide a routing layer so you can handle perform ajax (or anything else you want to do) when a link is clicked. The best bit is that if JavaScript, or pushState, isn't available the links can still point to real pages on your server