PingCAP CTO here, thanks for these comments! We highly appreciate all the feedback!
First, it’s true that the current setup/deployment of TiDB is not easy. This is something that we're making serious moves to improve. For example,
A. We provide Ansible playbooks to simplify the deployment and rolling upgrade for on-prem users;
B. We built and open-sourced TiDB Operator (https://github.com/pingcap/tidb-operator) to enable TiDB on Kubernetes. We are working on a fully managed service in the public cloud (coming soon). Whether it is one binary or multiple binaries, it’ll be all transparent at the user level;
C. We are improving the default or self-adaptative parameters and are continuously refining the configuration process;
D. We are also trying to reduce the number of components. For example, the new version of CDC is implemented directly inside TiKV.
E. We are developing TiOps tools in a single binary to improve the operating and maintaining experience of the cluster.
A fair amount of customers around the world are using TiDB in their production environments and we are making sure they get our help when needed in the setup so it would not be a deal-breaker.
Second, TiDB’s multiple-component or highly-layered architecture is challenging for deployment but the benefits are also obvious:
A. The separation of the storage and computing layers makes it flexible and agile to scale/upgrade each layer as needed. Different layers need different types or different number of hardware resources. If the computing resources become the bottleneck, users can scale the SQL layer by adding more TiDB instances in real-time; if the bottleneck is the storage layer, they can easily add more TiKV instances to increase the storage capacity.
B. As is known to many that we have donated TiKV to CNCF last year. We are fully committed to the open-source community and would like to see TiKV be the building block and foundation of the next-generation infrastructure. For example, we are happy to see some community users sit Redis on top of TiKV, and we ourselves built the TiSpark (https://github.com/pingcap/tispark) connector to run Apache Spark on TiKV.
PingCAP CTO here, I really enjoy the cooperation with Kyle this two months. I want to talk a little bit about my feelings.
Before the "official" Jepsen test, TiDB has already written its own Jepsen test cases for a long time. Our past Jepsen test code has become a good starting point in this test. In addition, as a result, the test case written by the database developer will inevitably have some blind spots. This is even if you have your own Jepsen test passed, I still recommend you to do the official test by Kyle's team.
In the process of testing, we have no way to influence what Kyle tests. Usually, only Kyle will tell us when abnormal behaviour occurs. Our main job is to analyze the cause and then fix it.
Here are some tips for recommending friends who want to do Jepsen test in the future:
1. Carefully check your documentation before testing to make sure it is consistent with your database implementation.
2. It is best for someone on the team to understand and build your own Jepsen test environment internally and test parallelly with the Kyle team.
3. Simplified deployment process is very helpful
4. Be sure the version of your DB before testing. Usually, you can select one when starting the test and select one before the end.
I really enjoy working with Kyle. I would say Kyle has a good sense of humor :), and I hopes to have another chance to cooperate Kyle and his team in the future! Thank you.
TiDB developer here, we're working on columnstore for TiDB, will release in the near future. On the query optimizer side, I think in TiDB 3.0, TiDB's optimizer has given the best execution plan for most of the TPC-H (TPC-H 50G) queries. But we've never tested on TPC-DS yet.
TiDB developer here, TiDB Binlog is the offical CDC tool for TiDB, it used to be a part of TiDB Enterprise Edition.
The TiDB Binlog component is used to collect the transaction log of TiDB and synchronize it to the downstream in real time, such as TiDB/MySQL/Kafka. This component is similar in function to MySQL's master-slave replication. It collects the binlogs generated by each TiDB instance and sorts them by the time of transaction commit, and globally and orderly synchronizes the data downstream. With TiDB Binlog, data can be synchronized to other databases in real time, as well as TiDB incremental backup and recovery. As one of the core components of TiDB, TiDB Binlog has been running stable for a long time in the production environment of hundreds of users.
We are very happy to be able to open source this and help more users.
Very interesting project, the problem with Serverless service provided by different public cloud vendors is that programming and API are not uniform. I think Faast.js is on the right path to creating a unified interface for different Serverless services.
'This model is equivalent to the multiprocessor RDBMS—which also uses a single physical clock, because it’s a single machine—but the system bus is replaced by the network. In practice, these systems give up multi-region scale out and are confined to a single datacenter.'
I don't think so, although Percolaotor has a single point of physical clock, but it is easy to eliminate the SPOF problem by using high-availability algorithms such as Raft or Paxos (Just like the design in TiDB).
On the other hand, the strong consistency across multiple data centers, I think, without hardware devices such as TrueTime, it is difficult to have low latency and strong consistency while allowing writes happen in multiple data centers at the same time. The more realistic situation is that there is a primary data center (where both read and write occur), and multiple data centers guarantee high availability with strong consistency, we can achieve this goal by scheduling Paxos or Raft leader to the primary data center.
TiDB's transaction model is different from Cockroach, and it's highly layered, you can use tikv(the underlying storage layer of tidb) as a key-value database without sql for better performance.
Alibaba is one of the biggest IT company in China, but I can't see any respect to the engineers. These engineers who was fired just wrote some automatic scripts to simulate mouse clicking, hardly a hack...but Alibaba's HRs didn't do any research, and fired them in 2 hours. I don't think it's fair.