100%, and we have all of those things. Canary acts as the last line of defence, and honestly, when Canary detects and rolls back, it is already an incident that is being auto-mitigated with a limited blast radius.
To reduce the potential blast radius, we are working on a cohort-based canary, which will allow us to validate against a minimal, stable subset of traffic with the desired properties.
TLDR:
A DNS automation bug removed all the IP addresses for the regional endpoints. The tooling that was supposed to help with recovery depends on the system it needed to recover. That’s a classic “we deleted prod” failure mode at AWS scale.
I love actors as a concept and I heard some large companies (Expedia) implemented large parts using them.
But I also saw how hard it is to understand a large system that built using actors. It is just hard to comprehend all the communication pathways and what happens in the system.
Some things to consider:
1) The company needs a way to weed out folks who are net negative. In general, if someone is not playing their part, there should be a mechanism to evict if up-leveling fails.
2) The company needs a way to distribute incentives (bonuses) as fairly as possible.
That is very interesting. As CRDB user, I priced Spanner (had to do some estimates during load testing), and Spanner came 3 times more expensive includign our eng salary to run CRDB
Fascinating.
How does streets align on the Chinese maps though? If random offsets are added, nothing should align. Or 500 meters is not large enough offset to create a mess?
What I mean, arrears that are parallel in reality, may not be parallel on a map.
I did some math. Assuming a human male needs around 3000 KCal per day, it sums up to around 1,095,000 KCal per year. Potato is one of the most calory dense, as 1 acre produces 17.8 M KCal (found online).
So, one person will need 1 Acre (43560 sq feet) / (17.8 / 1.095) = 2679.6741573 sq feet of potatoes. Which will be 248.8872 sq meters (10 by 20 meters field).
It is either my math is wrong, or article is wron by order of magnitude
To reduce the potential blast radius, we are working on a cohort-based canary, which will allow us to validate against a minimal, stable subset of traffic with the desired properties.