But the thing that doesn't get much discussion in public is the drama that happens from upgrades when behavior changes WRT file systems, kernels, device drivers, CPU frequency governors, etc.
It took a few weeks to figure that out for several of my servers. So I am slow to upgrade things once that gets stable -- and it has been stable for my servers for about 6 months.
Author here. The interesting point was that RSS is much too large with glibc malloc + RocksDB/MyRocks. The performance differences aren't a big deal, but OOM because RSS is 2X or 3X larger than it should be is a big deal. Of course, that problem is workload specific, but RocksDB/MyRocks are a workload on which I focus.
I lurk on email lists for RocksDB and OOM from glibc malloc is a problem for users -- the solution is easy, use jemalloc or tcmalloc.
I use the latest LTS release of Ubuntu, which is Ubuntu 22.04, and the library versions are what Ubuntu provides. So these are the libraries that most of us use when using Ubuntu LTS. Regardless, I am happy that tcmalloc and jemalloc continue to improve.
Occasionally I get "your version of X is too old" when I publish results for X, and I have had mixed results in the past when doing even more work to test the newer thing. In this case I am not going to put in that extra time. But in a few years I might revisit the topic and get newer versions of X, of course, there will be feedback that the version of X I test in 2028 is from 2026 and the story repeats.
Thanks for doing the sensitivity analysis. I struggle with that issue a lot -- how can I shorten my benchmark duration (use less data, run for less time) so I can get more work done.
This post inpsired a lot of great discussion about the Spanner family of distributed databases. Someone asked about TiDB there, but there was no follow up. Can someone answer that question?
Yes, context is everything when looking at benchmark results. I am showing the worst-case for this performance regression. Thanks for mentioning that. Note, that is my blog.
I also showed the impact for IO-bound workloads, and even there the regression is larger than I want. But not as bad as the in-memory workloads.
It took a few weeks to figure that out for several of my servers. So I am slow to upgrade things once that gets stable -- and it has been stable for my servers for about 6 months.