Yes, I had backups everywhere. Across providers, in different countries. But I built a system tied to my AWS account number, my instances, my IDs, my workflows.
When that account went down, all those “other” backups were just dead noise encrypted forever.
Bringing them up to the story only invites the 'just use your other backups' fallback, and ignores the real fragility of centralized dependencies.
It is like this: the UK still maintains BBC Radio 4’s Analogue Emergency Broadcast—a signal so vital that if it’s cut, UK nuclear submarines and missile silos automatically trigger retaliation. No questions asked. That's how much stakes they place on a reliable signal.
If your primary analogue link fails, the world ends. That's precisely how I felt when AWS pulled my account, because I’d tied my critical system to a single point of failure. If the account was just Read only, i will waited because i could have access to my data and rotated keys.
AWS is the apex cloud provider on the planet. This isnt about redundancy or best practices.
it's about how much trust and infrastructure we willingly lend to one system.
Remember that if BBC Radio 4 signal get to fail for some reasons, the world will get nuked, only cockroaches will survive… and your RDS and EC2 billing fees.
Thank you for your concern, and I appreciate the nuance in your take.
Yes, it is totally possible that AWS monitors blogs and forums for early damage control, like your company does.
But we shouldn’t paint it like I was bailed out by some algorithmic PR radar and nothing else.
Let’s not fall into the “Fuk the police” style of thinking where every action is assumed to be manipulation. Tarus didn’t reach out like a Scientology agent demanding I take the post down or warning me of consequences.
He came with empathy, internal leverage, and actually made things move.
When before i read Tarus email, i wrote in Slack to Nate Berkopec (puma maintainer): `Hi. AWS destroyed me, i'm going to take a big break .`
Then his email reset my cortisol levels to acceptable level.
Most importantly, this incident triggered a CoE (Correction of Error) process inside AWS.
That means internal systems and defaults are being reviewed, and that’s more than I expected. We’re getting a real update, that will affect cases like mine in the future.
So yeah, it may have started in the visibility layer, but what matters is that someone human got involved, and actual change is now happening.
"If you want your paperwork processed in Morocco, make sure you know someone at the commune, and ideally have tea with their cousin."
Yes, it works, but it shouldn’t be the system.
What happened with AWS isn’t a clever survival tip, it’s proof that without an account manager, you are just noise in a ticket queue, unless you bring social proof or online visibility.
This should have never come down to 'who you know' or 'how loud you can go online'.
It a big luck that i'm speaking in english and have online presence, what if i was ranting in French, Arabic, or even Darija in Facebook. Tarus will have never noticed.
The AWS employee actually contacted me before my blog post even reached three digits in views. So no, it wasn’t PR-driven in the reactive sense.
But here’s what I learned from this experience:
If you are stuck in a room full of deaf people, stop screaming, just open the door and go find someone who can hear you.
The 20 days of pain I went through, it wasn’t because AWS couldnt fix it.
It’s because I believed that one of the 9 support agents would eventually break script and act like a human. Or that they get monitored by another team.
Turns out, that never happened.
It took someone from outside the ticketing system to actually listen and say: Wait. This makes no sense.
The person paying the account is not the author, i'm.
What happen is that the person paying for the account had to settle an invoice of many thousand of dollars.
They offered me AWS gift cards,to send me electronics and they will pay for it in parts.
They lost lot of money because of crypto collapse. So i accepted their solution to pay for my OSS usage for few months.
That like if i was going to pay for your rent for 1 year. You don't pay, while i don't have to pay 3-4 years of your rent at once.
What happen, is that AWS dropped a nuclear bomb in your house, in the middle of the month .. then tell you later that it was about payment.
If they told me in the first email it was about the payer, i will have unlinked and backuped.
Yeah, it's fishy. I never claimed the Java theory is confirmed, just that it’s what an insider told me after the fact.
They said a dry-run flag was passed in the --gnu-style form, but the internal tool expected -dry, and since Java has no native CLI parser, it just ignored it and ran for real.
Supposedly the dev team was used to Python-style CLIs, and this got through without proper testing.
The idea of a rogue AWS team running a deletion script without oversight should sound ridiculous. At a company that size of AWS, you will expect guardrails, audits, approvals.
But here is the thing: no one from AWS has given me an official explanation. Not during the 20-day support hell, not after termination, not even when I asked directly: “Does my data still exist?” Just a slow drip of templated replies, evasions, and contradictions.
An AWS insider did reach out claiming it was an internal test gone wrong, triggered by a misused --dry flag and targeting At a company that size, you'd expect guardrails, audits, approvals low-activity accounts.
According to them, the team ran it without proper approval. Maybe it is true. Maybe they were trying to warn me.
Maybe its a trap to get me to throw baseless accusations and discredit myself.
I'm not presenting that theory as fact. I don’t know what happened behind the wall.
What I do know is:
- My account was terminated without following AWS’s own 90-day retention policy
- I had a valid payment method on file
- Support stonewalled every direct question for 20 days
My post is not about backup strategy, it’s about what happens when the infrastructure itself becomes hostile, and support throw you from one team to another. AWS didn't just delete files.
They gaslit me for 20 days while violating their own stated policies.
I dont disagree with your broader point—centralizing everything in one provider is a systemic risk.
The architecture was built assuming infrastructure within AWS might fail. What I didn’t plan for was the provider itself turning hostile, skipping their own retention policy, and treating verification as a deletion trigger.
I did have backups. Multi-region. Redundant. I followed AWS’s own best practices to the letter.
The only failure I didn’t plan for? AWS becoming the failure.
The provider nuking everything in violation of their own retention policies. That’s not a backup problem, that is a provider trust problem.
The reason i did not kept a local copy, was that i formatted my computer after a hardware failure, after the nurse dropped the laptop in the hospital i was on. Since i have a AWS backup, i just started with a fresh OS while waiting to get discharged to return home and redownload everything.
When i returned 6 days days later, the backup was gone.
Not AI-generated. Not everyone is born writing flawless English.
If it sounds like an LLM, maybe it is because people like me had to learn how to write clearly from LLMs because English is not our first language.
I could’ve written in my native tongue, but then someone else will have complained that not how english is structured.
Also, the story is real. Just because it is well-structured doesn't mean it's fiction. Yes, i used AI to resort it, but i can assure you that no AI will generate the Piers Morgan reference.
Hi,
The infrastructure was not complex at at all, i can transfer it in 1 day.
I was hospitalized, in another city, with all the computer at home, and locked behind 2FA.
They send me is on notice on Thursday, by Monday evening, all access was revoked.
For weeks i asked for a readonly access to my data, then they could take anytime they want to verify, they refused.
And he more i ask about my data, they more they avoid to speak about it.
Think about it , you could be sick, on a trip, having jetlag, in some festival, getting married... by the time you are back online, the delay was gone.
In fact, I even added an em dash shortcut to my Logitech POP keyboard. That’s dedication. See? — — — — —
I’m not just writing like this—I’ve industrialized it.
If it sounds like a chatbot, maybe it's because I have spent too much time training them and not enough time pretending to be a chaos monkey in HN threads.
Appreciate the feedback though.
Now excuse me while I reboot into Windows just to add and to my custom shortcuts.
I have still got one more slot—what do you think I should add?
Go wild. Bonus points if it makes someone uninstall the gem out of pure emotional confusion.
- Token buckets: Proactive rate control (smooth request throttling)
They're complementary patterns. I actually have a rate limiter gem with token buckets, sliding windows, and distributed rate limiting, but I will open source it when my ship lands on a planet or a moon. Still navigating hyperspace.
Your 2 space credits are worth their weight in dilithium.
Funny thing is the gem originally started exactly like that, pure explicit constructors. Then I wrapped it with the DSL sugar but the explicit approach never lost its capability. Both paths lead to the same destination.
I actually loved this suggestion so much that I added a whole section called "Captain Byroot's Guide to Explicit Circuit Construction" to the docs: