That's the reason why the default maximum edit distance of SymSpell is 2.
Now, all your 6 out of 6 examples are chosen from that 1.1% margin that is not covered by edit distance 2, presenting a rather unlikely high amount of errors within a single word.
The third-party SymSpell port from Justin Willaby, which you were using for benchmarking, clearly states that you need to set both maxEditDistance and dictionaryEditDistance to a higher number if you want to correct higher edit distances. That you neither used nor mentioned. This has nothing to do with accuracy; it is a choice regarding a performance vs. maximum edit distance tradeoff one can make according to the use case at hand.
Author of SymSpell here. Congrats on the launch of Lexiathan.
Unfortunately, the comparison of Lexiathan vs. Symspell on your website regarding accuracy is misleading.
1. SymSpell has two parameters to control the maximum edit distance. Once you set both to 3, then also terms with an edit distance of 3 are accurately corrected:
SymSpell accurately corrects all of your examples if used properly with the correct parameters and dictionary.
Apart from that, your methodology of comparing correction accuracy by cherry-picking specific terms without statistical significance, where your product seemingly performs better, is questionable.
One would use large public corpora to measure the percentage of accurately corrected terms as well as the percentage of false positives.
Because SymSpell is Open-Source, everyone can integrate it into their applications for free, modify the code, use different dictionaries in various languages, or add terms to existing ones.
Can the index size exceed the RAM size (e.g., via memory mapping), or are index size and document number limited by RAM size?
It would be good to mention those limitations in the README.
The most widely used DHT is Kademlia from Petar Maymounkov and David Mazières.
It is used in Ethereum, IPFS, I2P, Gnutella DHT, and many other applications.
SeekStorm does currently not use io_uring, but it is on our roadmap.
Challenges are the cross-platform compatibility. Linux (io_uring) and Windows (IoRing) use different implementations, and other OS don't support it. There is no abstraction layer over those implementations in Rust, so we are on our own.
It would increase concurrent read and write speed (index loading, searching) by removing the need to lock seek and read/write.
But I would expect that the mmap implementations do already use io_uring / IoRing.
Yes, lazy loading would be possible, but pure RAM access does not offer enough benefits to justify the effort to replicate much of the memory mapping.
For the latency benchmarks we used vanilla BM25 (SimilarityType::Bm25f for a single field) for comparability, so there are no differences in terms of accuracy.
For SimilarityType::Bm25fProximity which takes into account the proximity between query term matches within the document, we have so far only anecdotal evidence that it returns significantly more relevant results for many queries.
Systematic relevancy benchmarks like BeIR, MS MARCO are planned.
In SeekStorm you can choose per index whether to use Mmap or let SeekStorm fully control Ram access. There is a slight performance advantage to the latter, at the cost of higher index load time of the former.
https://docs.rs/seekstorm/latest/seekstorm/index/enum.Access...
The benchmark should be fairly fair, as it was developed by Tantivy themselves (and Jason Wolfe). So, the choice of corpus and queries was theirs. But, of course, your mileage may vary. It is always best to benchmark it on your machine with your data and your queries.
Currently, you can choose between tokenizers with or without folding. But configurability per language or full customizability of the folding logic by the user is a good idea.
The code for the distributed search cluster is not yet stable enough to be published, but it will be released as open-source as well.
As for shared storage, do you mean something like NAS or, rather Amazon S3?
Cloud-native support of object storage and separating storage and compute is on our roadmap. Challenges will be maintaining latency and the need for more sophisticated caching.
Its not just about speed. Speed reflects efficiency. Efficiency is needed to serve more queries in parallel, to search within exponentially growing data, with less expensive hardware, and fewer servers, consuming less energy. Therefore the pursuit for efficiency never gets outdated and has no limit.
PostgreSQL is an SQL database that also offers full-text search (FTS), with extensions like pg_search it also supports BM25 scoring which is essential for lexical search. SeekStorm is centered around full-text search only, it doesn't offer SQL.
Performance-wise it would be indeed interesting to run a benchmark. The third-party open-source benchmark we are currently using (search_benchmark_game) does not yet support PostgreSQL. So yes, that comparison is still pending.
Yes. We waited long for AOT compilation to become mature, to remove the need for the user to install the .Net framework. But two years ago when we decided to switch, we still couldn't just get the AOT compilation of our codebase to work without changes (perhaps it was somehow possible, but the available documentation was not very verbose about this). Also, there is still a performance gap. Of course, this doesn't matter for most of the applications, where the completeness and consistency of the framework, and the number of programmers fluent in that language might matter more. But for a search server, we needed to carve out every inch of performance we could get. And other benchmarks seemed to echo our experience: https://programming-language-benchmarks.vercel.app/rust-vs-c...
The 2-4 speed ratio was not meant to denounce C#, which is a great language I loved to program in for over two decades, coming from Delphi.
Unfortunately, C# has not a complete SIMD support.
See our request to support the SSE4.2 _mm_cmpistrm instruction https://github.com/dotnet/runtime/discussions/63332, which we required for a vectorized intersection between two sorted 16-bit arrays.
We did the switch from C# to Rust not light-minded, as the cost of porting a fairly large codebase is time-consuming.
We just wanted to share our experience for our specific task, not as a general statement.
That's the reason why the default maximum edit distance of SymSpell is 2.
Now, all your 6 out of 6 examples are chosen from that 1.1% margin that is not covered by edit distance 2, presenting a rather unlikely high amount of errors within a single word.
The third-party SymSpell port from Justin Willaby, which you were using for benchmarking, clearly states that you need to set both maxEditDistance and dictionaryEditDistance to a higher number if you want to correct higher edit distances. That you neither used nor mentioned. This has nothing to do with accuracy; it is a choice regarding a performance vs. maximum edit distance tradeoff one can make according to the use case at hand.
https://github.com/justinwilaby/spellchecker-wasm?tab=readme...
pronnouncaition IS within edit distance 3, according to the Damerau-Levenshtein edit distance used by SymSpell. The reason is that adjacent transpositions are counted as a single dit. https://en.wikipedia.org/wiki/Damerau%E2%80%93Levenshtein_di...