I thought about TOML for a while. Went with JSON because I just wanted to start working.
Like I said in one of the comments, final decision has not been made yet. JSON is not written in stone, plus ffcms could support more than one language, e.g. JSON and TOML.
Aaand of course, thanks for the advice. Personally, I've never been in a situation where `filter_complex` caused problems. But, I've never done some weird filtering there too, so you may be right.
I'm not saying that now, with ffcms, you should write JSONs that produce an A4 ffmpeg command. If you're doing so, something is not right.
For now, it's just a tool that helps me with timelapse creation for my fiancée. Let's see how it evolves.
There is no restriction on using okon only for HIBP passwords. Actually, the goal is to handle hashes, no matter where they come from. User may have reasons to update db frequently.
Yes, you're right that sorting data using B-tree is slower than other methods. But I don't use it only for sorting. It's not like 'create B-tree and forget about it'. You create the tree from initial data, and if you need to insert any more hashes there, you can do it really quickly, without a need to sort everything again.
I agree again. If I'd create okon only for my home usage, I'd probably use a real database. But I didn't want to depend on anything. I wanted to create a library that can be used by any program and just works, without forcing user to install anything else. That's why I sort the data on my own, and that's why I implemented B-tree on my own. Everything is in okon's codebase.
Thanks (:
About the Postgres, I wanted to create a library and a CLI without dependencies. I wanted them to be a complete tools for doing this one thing. Tools that you can just grab and use, without installing anything.
True, but I wanted to create a library and a CLI without dependencies. I don't want to force an end user to install and use a database (even under the hood).
I don't think that expecting that an end user has installed a whole database toolchain is a good solution.
The purpose of the tool and the library is to be a complete solution for this one thing. The user can grab the binary and just use it, without installing/using any database under the hood.
As I said in the article, I rejected binary search and similar approaches on purpose.
In general, I rejected all approaches that operates on a sorted file. That's simply because keeping the file sorted is slow, when you need to insert more data in the future. With B-tree approach, you have fast searching, as well as, inserting. That's the reason.
You're right. As I said in some comment up there, the benchmarks are bad.
I benchmarked the best and the worst case in scope of the original file, so I look up for the first and the last hash.
I totally missed that if I look for the same hash over and over again, I'd end up reading the same B-tree files parts, so they can be easily cached. That's probably the reason it seemed so fast.
I'll rewrite benchmarks and update the results.
About the Bloom filter, I thought about that but I wanted a solution that is 100% correct. The filter doesn't guarantee that.
About the sorting. I wanted to implement a stand-alone library that does everything for you and that is cross-platform. That's why everything is implemented, without dependencies/usage of other tools.
That's exacly why I publish my pet projects. To learn from others.
You're right. The benchmarks are bad.
I benchmarked the best and the worst case in scope of the original file, so I look up for the first and the last hash.
I totally missed that if I look for the same hash over and over again, I'd end up reading the same B-tree files parts, so they can be easily cached. That's probably the reason it seemed so fast.
Like I said in one of the comments, final decision has not been made yet. JSON is not written in stone, plus ffcms could support more than one language, e.g. JSON and TOML.