Glad to hear that :D One of my main goals was to show the entire progression of how I approached the problem, instead of just saying "here's the final result".
(replying with more details, because I think it's pretty cool, and I'll definitely try this out)
The linked item is to https://blog.mro.name/2022/08/pwned-diy/, where the author converts the passwords list to a CDB (Constant DataBase) file, a simple, fast lookup database format created by D. J. Bernstein.
I actually did try this, but I got impatient and stopped it after 30 minutes or so, and a ~20G database file.
For some reason, after that, `ls` in that directory (or even shell completion) would freeze the shell entirely, even after reboot. I eventually managed to delete the file with `rm db.sqlite` (no completion allowed), but even that took like 2 minutes.
I might try again with WAL enabled (based on my shell history, I also deleted a -journal file).
(author here) Hey, that's actually a great idea. I remember reading about a fast grep or wc implementation that did parallel reads.
Off the top of my head, I don't really know how to do that (aside from the naive way of using threads), I'm guessing readv[1] might be be useful? Will definitely look into it.
Since SQL is ever-so-slightly different across databases, I imagine trying to cover all of them as a single dev is a nightmare (especially if that's not the problem you're trying to solve).
I wrote my own query builder because I know for sure I'm only targeting SQLite. The second I need my feed reader library to work with another database engine I'm dumping my own for something more serious – either a full blown database abstraction layer like SQLAlchemy or Peewee (likely without the ORM part), or something simpler like PyPika or python-sql.[1]
> It's magic when it works but it suffers from the drawbacks noted elsewhere in the thread about traditional ORMS...
(Not that there are lots of other devs on my project at the moment, but)
I am managing this by aggressively limiting the size / features of my query builder, so it's (relatively) easy to understand just by looking at the code.
If it ever passes that point, it's probably time to switch to a "real" query builder / ORM.
http://web.archive.org/web/20170913234634/https://medium.com...
http://webcache.googleusercontent.com/search?q=cache:https:/...