We Salute You, S3 Team Member(gofundme.com)
gofundme.com
We Salute You, S3 Team Member
https://www.gofundme.com/we-salute-you-s3-team-member
6 comments
Light-hearted or not, this is pretty patronizing. I'm sure this person wants nothing more but to get on with their life.
We all have been there.
I remember removing a 10gig port channel (by a typo) on a regional switch. That was in 2012. The horror.
I remember removing a 10gig port channel (by a typo) on a regional switch. That was in 2012. The horror.
What does it say when the creator of the fundraiser hasn't even donated himself?
That person works at Amazon, they don't need a GoFundMe.
The Amazon post itself[0] identifies several other contributing factors beyond "Sysop Error". Root cause analysis would ask the following questions:
Why is a potentially risky operation being done manually in the first place?
Why was the human error of restarting a larger-than-usual percentage of servers not flagged for user confirmation before continuing?
Why was a restart of the indexing subsystem and the placement subsystem never tested in the past several years?
etcetera, etcetera.
In short, there were significant process failures that led to "S3 Team Member" even being in a position to accidentally cause this outage.
Perhaps whoever is behind this "light-hearted" pledge drive can't see beyond the human factor because they don't have experience at anything close to this scale, and systems many orders of magnitude smaller simply can't justify the sort of investment in extra belts and suspenders that is done on larger systems.
Any of the big hosts, including Amazon, invests quite a lot in infrastructure and tooling precisely to prevent or at least mitigate this type of error, but this particular corner of their system didn't have enough protections in place nor was the recovery process tested.
Those process failures are not "S3 Team Member"'s fault. At. All.
As far as I am concerned, this was a bug, and "S3 Team Member" should get a prize for finding it, however inadvertently. May I suggest some extra vacation time?
[0] https://aws.amazon.com/message/41926/