Hi HN, this is a series on a way to generalize consensus protocols and how to adapt them to existing systems.
This is part 7: Discovery and Propagation. With this, we complete everything about the protocol part of Generalized Consensus. There are a few more advanced topics to cover: They are needed for running in production.
One of our goals is to make this approach work for Postgres via Multigres. But the principles can be used to implement your own solutions also. This will make Durability and High Availability more accessible for users.
Hi HN, this is a series on a way to generalize consensus protocols and how to adapt them to existing systems.
This is part 6: It covers Revocation and Candidacy, the prerequisites for a leader change. We explain how to revoke previous leaderships and recruit for a new candidacy. It's a bit complex, but there are plenty of animations to help you along.
One of our goals is to make this approach work for Postgres via Multigres. But the principles can be used to implement your own solutions also. This will make Durability and High Availability more accessible for users.
Hi HN, this is a series on a way to generalize consensus protocols and how to adapt them to existing systems.
This is part 5: This one covers how to safely handle multiple coordinators racing to take action. We explore term numbers, coordinators as separate agents, and lock-free approaches.
One of our goals is to make this work for Postgres via Multigres. But the principles can be used to implement your own solutions also. This will make Durability and High Availability more accessible for users.
Hi HN, this is a series on a way to generalize consensus protocols and how to adapt them to existing systems.
This is part 4: This one has a nice animation that shows how requests get fulfilled.
One of our goals is to make this work for Postgres via Multigres. But the principles can be used to implement your own solutions also. This will make Durability and High Availability more accessible for users.
Hi HN, this is a series on a way to generalize consensus protocols and how to adapt them to existing systems.
This is part 3: Governing Rules. This is the foundation for the rest of the series. We'll be repetitively applying these rules to map out all parts of the protocol. Also, every system that I know of implicitly follows these rules.
One of our goals is to make this work for Postgres via Multigres. But the principles can be used to implement your own solutions also. This will make Durability and High Availability more accessible for users.
Hi HN, this is a series on a way to generalize consensus protocols and how to adapt them to existing systems. I've released two parts so far. They explain the problem and the requirements. You can follow the link to the second part from the first part.
One of our goals is to make this work for Postgres via Multigres. But the principles can be used to implement your own solutions also. This will make Durability and High Availability more accessible for users.
I oversaw this work, and I'm open to feedback on how things can be improved. There are some factors that make this particular situation different:
This was an LLM assisted translation of the C parser from Postgres, not something from the ground up.
For work of this magnitude, you cannot review line by line. The only thing we could do was to establish a process to ensure correctness.
We did control the process carefully. It was a daily toil. This is why it took two months.
We've ported most of the tests from Postgres. Enough to be confident that it works correctly.
Also, we are in the early stages for Multigres. We intend to do more bulk copies and bulk translations like this from other projects, especially Vitess. We'll incorporate any possible improvements here.
The author is working on a blog post explaining the entire process and its pitfalls. Please be on the lookout.
I was personally amazed at how much we could achieve using LLM. Of course, this wouldn't have been possible without a certain level of skill. This person exceeds all expectations listed here: https://github.com/multigres/multigres/discussions/78.
You won't believe what a nightmare it was to work with transactionless DDLs in MySQL. Transactional DDL will be a dream come true for Vitess: we can throw away all the hacks we had to do for MySQL's sake.
I also see such a clean 2PC API. This was another huge mess on the MySQL side.
The Vitess architecture was traditionally built in a database agnostic fashion. Most of its features should port smoothly over to Postgres.
The cool features that I can think of: A formal sharding scheme based on relational foundations, a fairly advanced query analyzer and routing engine capable of cross-shard functionality, HA and durability, abilities to reshard safely, seamless migrations, etc.
It will be a proxy layer. Since Vitess has proven itself with this approach, I don't see a reason to try anything drastically different.
As for Planetscale, I've posted my thoughts on X and LI :).
Hello HN, co-creator of Vitess here. The goal of Multigres is to build an open source adaptation of Vitess for Postgres.
In terms of project plan, we're still deciding. One option is to fork and retrofit. This will give us immedate results, but we'll be inherting quite a bit of legacy that we'll have to overcome later. The other option is to start with a clean slate and move pieces from the original Vitess code. This will ensure that Multigres has the Postgres DNA. We'll also avoid inheriting legacy features. Let us know your thoughts on this.
I'll try my best to answer any other questions you may have.
It's not just failure detection. A write to EBS is at least two additonal network hops. The first one is to get to the machine for the initial write, and the second is for that write to be propagated to another machine for durability. Multiply this by the number of IOPS required to complete a database transaction.
The way to reason about this is: If a customer had a huge business, what key would they shard by? Then choose that as the sharding key for all customers.
This is part 7: Discovery and Propagation. With this, we complete everything about the protocol part of Generalized Consensus. There are a few more advanced topics to cover: They are needed for running in production.
One of our goals is to make this approach work for Postgres via Multigres. But the principles can be used to implement your own solutions also. This will make Durability and High Availability more accessible for users.