Great article. A lot of engineers don't have personal experience with these kinds of network failures, so sharing stories of their consequences means more engineers can make informed (and conscious) decisions of how much risk can be tolerated for their applications.
One thing that you could gleam for this article-and I think that this is incorrect-is that the application or operations engineer is responsible for understanding the nuances of distributed systems. In my experience the number of people who are relying on distributed systems is much larger than the number of people who understand these issues.
So what we really need are systems we can build on whose developers understand how to build (and test!) the nuances of data convergence, consensus algorithms, split-blain avoidance, etc. We need systems to gracefully-and automatically-deal with and recover from network failures.
What kind of consistency do you expect to provide with this future syncing feature? I assume it will be eventually consistent. Is that right? How will conflicts be resolved?
The author says that different data stores are good at different things, so we need to use multiple data stores in our application. He proposes that a data service layer can abstract these implementations, making it easier to swap out data stores as needed. I think that's a good idea. Separating our applications from data store specifics is a big reason why we use ORMs and ODMs today.
However, I think there are challenges with this polygot data store architecture that he doesn't address. Each addition requires due diligence to understand its CAP trade-offs (which the author mentions briefly), scalability and performance characteristics, how to configure, etc. These are non-trivial concerns even for a single database. It's important to consider these challenges when building out a data store or data service.
I'd propose an architecture where the data services layer itself exposes different data models to the application, all of which are persisted in a single data store. Given that many data stores use a key-value store under the covers anyway, translating the specific data model down to a single, persistent data store would simplify operations while exposing the desired data model to the application. (As a caveat, this multi-model approach requires ACID transactions to ensure strict consistency when translating between data models.) This approach provides operational simplicity with just one data store and application efficiency by exposing the "right" data model API.
Full disclosure: I'm an engineer at FoundationDB, a database that provides ACID-compliant polygot data models on a single key-value data store (http://www.foundationdb.com).
Hey guys, I'm the author of the blog post. I'm happy to answer questions about performance results, argue about the necessity of acid transactions, or discuss anything else in the post.
I've been playing around with the idea of doing a couple of these posts - for example, I could do one with multiple datacenters. This would be a good scenario too.
Right. Most of the time you would simply retry the transaction - that's what FoundationDB's automatic transaction retry loop does. The first two tests I ran do that, but the third (foundationdb-append-noretry) does not.