Falsehoods Software Developers Believe About Event-Driven Systems(dimtion.fr)
dimtion.fr
Falsehoods Software Developers Believe About Event-Driven Systems
https://dimtion.fr/blog/falsehoods-event-driven/
4 comments
Isn't message ordering a fundamental property / guarantee of the underlying messaging system protocol?
https://en.wikipedia.org/wiki/Atomic_broadcast
Chalk me up to believing a falsehood, what's the failure case where such a protocol breaks down? Is this a "More than 50% of nodes in byzantine fault failure mode" situation, or a realistic scenario?
https://en.wikipedia.org/wiki/Atomic_broadcast
Chalk me up to believing a falsehood, what's the failure case where such a protocol breaks down? Is this a "More than 50% of nodes in byzantine fault failure mode" situation, or a realistic scenario?
I think the text is meant as a checklist you can use to apply to some system, not a list of impossibilities.
For example:
> Events will arrive in order, even if specified by the producer contract
This reminds me of a system I worked on. It used FIFO queues, but even with very low throughput we were getting out of order records on our database.
It turns out there was a rogue worker re-queueing some items into a non-FIFO standard queue that CAN deliver out of order stuff, and that was messing things up.
We were trusting the producer contract, which works well enough, but there were guarantees we needed to make on _OUR_ side to ensure proper ordering all the way through.
For example:
> Events will arrive in order, even if specified by the producer contract
This reminds me of a system I worked on. It used FIFO queues, but even with very low throughput we were getting out of order records on our database.
It turns out there was a rogue worker re-queueing some items into a non-FIFO standard queue that CAN deliver out of order stuff, and that was messing things up.
We were trusting the producer contract, which works well enough, but there were guarantees we needed to make on _OUR_ side to ensure proper ordering all the way through.
But it's not a useful checklist, particularly if it doesn't have examples. Your post is more informative than the whole article, because it's a concrete example of where an assumption went wrong.
I could say, "Falsehoods programmers believe about databases:"
And say
* Transactions are atomic
But it's not helpful without examples of when and why they're not atomic.
And it's not helpful to programmers to write their software as if transactions are not atomic. The ability to think about transactions as atomic is a fundamental of the database system.
A programmer is far more effective if they leverage the fact that their DB does have atomicity guarantees and write their software accordingly, rather than slowing everything down by trying to code around an assumption they are not, because perhaps in just the right failure mode or configuration of the right database product, they might not be.
That's why examples are so important, because going through this checklist in code review is a huge waste of time if it's not actually applicable. Being able to leverage assumptions about a system is a force-multiplier.
I could say, "Falsehoods programmers believe about databases:"
And say
* Transactions are atomic
But it's not helpful without examples of when and why they're not atomic.
And it's not helpful to programmers to write their software as if transactions are not atomic. The ability to think about transactions as atomic is a fundamental of the database system.
A programmer is far more effective if they leverage the fact that their DB does have atomicity guarantees and write their software accordingly, rather than slowing everything down by trying to code around an assumption they are not, because perhaps in just the right failure mode or configuration of the right database product, they might not be.
That's why examples are so important, because going through this checklist in code review is a huge waste of time if it's not actually applicable. Being able to leverage assumptions about a system is a force-multiplier.
I think the purpose is more, "you think that transactions are always atomic, but they might not be, so when you start using a new system, check to make sure". Everybody has a set of assumptions, and those assumptions aren't always true. Sometimes they are. So knowing "here's a place that the assumption is false" is interesting, but not necessarily helpful. Instead, these kinds of lists let you know which of the facts that you know are merely assumptions, so you can investigate if it's true or not in your particular case, and then decide if it matters.
Yep, you are right. Examples would go a long way. Now that I stopped to think about it, once you know these issues by heart, you kind of don't need the checklist anyway.
Could anyone suggest a more in-depth reading on this subject?
This surely merits another Message Duplication section.
1. Using a cliched headline style adds authority to your article.
In all seriousness, this article is just a list of unsupported assertions. I agree with many of them because of my own experience. But those with which I disagree, there is nothing to convince me. A much more useful article would explain nuance, risks, consequences, handling strategies.
For example, "Events will arrive in order". What causes that not to happen? Does the number of events, technology used affect it? What are the consequences of believing they will arrive in order, and then discovering they don't? What are the tradeoffs in designing the system to make it happen less frequently vs tolerating them arriving in any order?