Sorry if the post sounds too marketing-heavy. I'm sharing this case study because it addresses a common technical challenge: scaling a high-traffic application (Kwai, a video platform) from traditional MySQL to a distributed database system. It describes the journey from managing 300+ MySQL shards to consolidating into a single 400TB cluster using OceanBase, handling over 1 million QPS during peak loads. While the post is oriented towards promoting OceanBase, it resonated with developers because it provides specific technical details about real-world scaling problems, architecture decisions, and performance metrics that many engineering teams face when growing beyond traditional MySQL setups.
It has a lot of use cases in the mining industry, such as monitoring mine water, ventilation systems, fluids in tanks and pipes, distance and angle measurements, remote operation of actuators and equipment, environmental monitoring, etc.
Disclaimer: PingCAPer here.
Thanks for your attention to PingCAP and TiDB. It’s true that we are on HN front page for many times - that’s incredible recognition for the value of our technical content, and we are humbled and thankful. We take great pride in our work, and have always committed to valuable, accurate and insightful content.
As for referring our PVLDB as “the first industry paper”, thanks @karsinkk for pointing out. We admit that the original description was inaccurate and just updated it to be “This is the first paper in the industry to describe the Raft-based implementation of a distributed Hybrid Transactional/Analytical Processing (HTAP) database.”
However, we have never claimed ourselves as “the best, the fastest.” If you find such occurrences and think they are inaccurate, please do share with us and we will correct the same ASAP. As pointed in https://pingcap.com/blog/9-why's-to-ask-when-evaluating-a-di..., “There's no single technology that can be the elixir to all your problems. The database realm is no different. If your data can fit on a single MySQL instance without too much pressure on your server, or if your performance requirement for complex queries isn't high, then a distributed database may not be a good choice. Choosing to use a distributed database typically means additional maintenance cost, which may not be worthwhile for small workloads. “
I am wondering have you considered the open source NewSQL database solutions like TiDB, CockroachDB? They have the best of both traditional RDBMS and NoSQL and would be the perfect choice to the hyper-growth scenarios.
TiDB could be considered as a scale-out MySQL and CRDB a scale-out Postgres. They are all Spanner-inspired solutions that can help avoid manual sharding.
For the impatient: “The etcd key-value store is a distributed database based on the Raft consensus algorithm. In our 2014 analysis, we found that etcd 0.4.1 exhibited stale reads by default. We returned to etcd, now at version 3.4.3, to investigate its safety properties in detail. We found that key-value operations appear to be strict serializable, and that watches deliver every change to a key in order. However, etcd locks are fundamentally unsafe, and those risks were exacerbated by a bug which failed to check lease validity after waiting for a lock.”
Thanks for the comment! CockroachDB is now under BSL and CCL licenses. Different users have different opinions but you are right, we need to be accurate about it. Please give me some time to consult our user and see whether we can update this case study.
Want to make your own Golang apps run in a browser? Here’s what we learned ...
We had a lot of fun—and learned a lot—from our little adventure at TiDB Hackathon 2019. If you want to make your own Golang applications run in a browser, here are some suggestions:
As [WebAssembly Weekly - Issue #100](https://wasmweekly.news/issue-100/) mentioned, "WebAssembly support for Go applications is very much in its infancy.” For example, Golang hasn't fully supported WASI, and goleveldb doesn’t support Wasm/js. Be discreet before you go too deep, and clearly understand the use cases for your application.
Make sure your application doesn't have third-party platform-specific dependencies that can’t be compiled to Wasm.
Browsers don’t allow port listening and file operations, so you’ll have to work around those limitations.
- TiDB-Wasm includes TiDB server only. TiDB is the stateless MySQL component of the TiDB platform (check other components in the TiDB Architecture). Compiled to Wasm, this tool is only suitable for experimenting with SQL and testing out TiDB compatibility with MySQL queries.
- The Wasm binary file is close to 70 MB and compressed to 13 MB. The database and the data you create in TiDB-Wasm are written to memory, and therefore consume a lot of memory resources.
- TiDB-Wasm does not support most of the web browsers on mobile devices yet.
- The tool currently cannot demonstrate multi-node TiDB deployments. This means you can’t play with TiDB’s titanium-solid resiliency, test doing rolling updates, or online scale-out and scale-in.
Thanks to Wasm (https://webassembly.org/) for making it possible to build desktop-quality applications in the web browser.
Thanks to the Golang for supporting Wasm, and making it possible to run performant Golang applications or even databases in the web browser.
Thanks to the Ti-Cool team in the TiDB community for contributing TiDB-Wasm in the 2019 TiDB Hackathon, making running TiDB in local model with by embedding GoLevelDB in the web browser possible. The team members are currently working on a technical blog to share how they made it. Stay tuned!