I agree 100% with everything you wrote. It was very surprising to me that every transaction and every write operation blocks the whole database and not only the table it's performed upon (it makes sense since it's all a single file, but still).
The only workaround for this is splitting your main database into multiple databases, but this bleeds into your application logic and gets messy quickly. If you are in this position, it's best to switch to Postgres as you did!
Don't get me wrong, the PR review is still rigorous and can take weeks. But everybody I interacted with during the process was very supportive and helpful, so the overall experience was great!
While this blog post was at the #2 spot on HN it was getting 0.138 rps, so yeah 14k RPS is A LOT!
That said it's still good to know when your tools will break! As far as I understand it's not really about RPS but more about constant write operations preventing the WAL file from being flushed. You can reproduce this issue with a lot fewer RPS.
From what I know, the journal_size_limit PRAGMA still affects WAL mode, but it doesn't solve the issue of the WAL file potentially growing uncontrollably. Am I missing something?
I've heard this is a potential issue but I've never encountered it. Do you know about the experimental WAL2 branch[0] that splits the WAL file into two to circumvent this problem? You'd have to compile SQLite yourself from the WAL2 branch to try it out, but it might be worth it at your scale.
Python is much slower than Go, so I don't think we could get 14k RPS as easily with Django, but I do have to see if I can reproduce the problem in Django. Topic for a future blog post!
Thanks for sharing this, I really appreciate knowing where the limits of SQLite are!
I'm not a Django core dev, but I have managed to get my changes merged into Django already (the transaction_mode setting in 5.1 was my contribution).
Carlton does seem to be onboard with my idea[0], so I'm optimistic that we can make it happen. Comments like yours will help me make my case so thank you for that!
Hey HN! I'm the author of the blog post. I didn't mention this in the post, but I'll try to merge some of these settings into Django by making them the new default or the default for new projects created with the start project command.
Any feedback on all of this is greatly appreciated!
You've missed: pdm, uv, pip-tools, pipx, rye, and probably some others.
Only pdm and poetry generate cross-platform lock files by default as far as I know, but there are a lot of people trying to solve this problem right now.
It's not an easy problem to solve. Python's package management predates package managers from most other programming languages and Python itself predates Linux. There is a lot of baggage so change is very slow.
I'm not sure if you can compare a project that came out of a university and got adopted by Apple with a project developed by a VC backed company with no revenue. I'm sure Charlie has the best intentions with both ruff and uv, but we have no idea how this is going to play out.
If they stop development on Ruff today, Ruff won't be useful anymore after a few new Python releases. If the maintainers of the tools that Ruff is replacing stop maintaining them, the whole Python community will be in a really bad place. So, there is reason to be cautious. Astral being transparent on how they plan to make money would be very helpful.
What they are doing is already discouraging people from contributing to the projects their tools are replacing[0]. If they go out of business and stop supporting their tools, it might leave the Python community in a bad place.
I hope they answer this ASAP. People in the Python community are already concerned that Astral is following the Embrace, extend, and extinguish playbook.