Major lesson from when I worked on Google Search indexing is that queues have a lot of hidden complexity and can make your outages much longer than they need to be. We had a big project to get rid of a bunch of queues by just scaling up our synchronous backends and making them faster.
Interesting. We run pgbouncer via kubernetes so it was straightforward to make multiple pgbouncer processes on one machine. Also straightforward to get them running on multiple machines, which helps because we run on Azure and they like to cause rolling outages across our fleet via VM maintenance...
We run 1000s of machines in Azure. It's garbage. Very few features work. Nodes are always having strange issues, especially on the networking side. And the worst part is that Azure support has 0 interest in actually debugging things. We just got out of an outage today caused by the insanely slow SSDs that they attach to their postgres dbs by default.
> At some point, you will have many teams. And one of them _will not_ be able to validate and accept some upgrade. Maybe a regression causes something only they use to break. Now the entire org is held hostage by the version needs of one team. Yes, this happens at slightly larger orgs. I've seen it many times.
The alternative of every service being on their own version of libraries and never updating is worse.
It was low. I got a bump to 90k that year, then 130k when I jumped companies, which I thought was a mind boggling amount. Do entry level devs even get out of bed for $130k these days?
At Google, they found that engineers L5 and above got more work done with RTO, and engineers at L4 and below got significantly less work done. WFH is great but it doesn't work for fresh engineers (who are often the most gung-ho about it as well).
We use Python to generate these configs at my work. Ends up working out pretty well. I previously worked on the biggest deployment of gcl (the inspiration for KCL, Jsonnet) at Google and it was a giant nightmare and the cause of many outages.