Great writeup (including the human cost, e.g. loss / lack of sleep, which in my experience has a huge impact on complicated incident resolution).
Here’s what jumped out at me: “The new account was created in our database with a null value in the URI field.”
Almost every time I see a database-related postmortem — and I have seen a lot of them — NULL is lurking somewhere in the vicinity of the crime scene. Even if NULL sometimes turns out not to be the killer, it should always be brought in for questioning.
My advice is: never rely on NULL as a sentinel value, and if possible, don’t allow it into the database at all. Whatever benefits you think you might gain, they will inevitably be offset by a hard-to-find bug, quite possibly years later, where some innocuous-seeming statement expects either NULL or NOT NULL and the results are unexpected (often due to drift in the semantics of the data model).
Although this was a race condition, if the local accounts and the remote accounts were affirmatively distinguished by type, the order of operations may not have mattered (and the account merge code could have been narrowly scoped).
An interesting story, whether fact or fiction. The difficulty of receiving messages as “Brenda” from people in various forms of real-life distress and not being able to respond to them reminded me of playing the game “Eliza,” possibly the most un-Zachlike-like game ever released by Zachtronics: https://www.zachtronics.com/eliza/
The bit about monetizing one’s influence / social “capital” reminds me of Cory Doctorow’s novel _Down and Out in the Magic Kingdom_ (which is a great read regardless).
I'm a technologist, but I can't help but be reminded of the novel _This Perfect Day_ by Ira Levin here.
(Especially once you replace the question "do I have enough money to walk out the door with this item?" with "is my government-calculated social reputation score high enough to walk out the door with this item?")
Airline companies don't want to buy aircraft because their highest-probability outcome is to declare bankruptcy (which happens about every 10 years for most major airlines in the US, it seems). Far simpler to tear up your lease than to dispose of large, expensive assets like aircraft as part of the bankruptcy.
Here’s what jumped out at me: “The new account was created in our database with a null value in the URI field.”
Almost every time I see a database-related postmortem — and I have seen a lot of them — NULL is lurking somewhere in the vicinity of the crime scene. Even if NULL sometimes turns out not to be the killer, it should always be brought in for questioning.
My advice is: never rely on NULL as a sentinel value, and if possible, don’t allow it into the database at all. Whatever benefits you think you might gain, they will inevitably be offset by a hard-to-find bug, quite possibly years later, where some innocuous-seeming statement expects either NULL or NOT NULL and the results are unexpected (often due to drift in the semantics of the data model).
Although this was a race condition, if the local accounts and the remote accounts were affirmatively distinguished by type, the order of operations may not have mattered (and the account merge code could have been narrowly scoped).