(Hopping in here because the discussion is interesting... feel very free to ignore.)
Thanks for writing this up! It was a very interesting read about a part of networking that I don't get to seriously touch.
That said: I'm sure you guys have thought about this a lot and that I'm just missing something, but "why can't every proxy probe every [worker, not application]?" was exactly one of the questions I had while reading.
Having the workers being the source-of-truth about applications is a nicely resilient design, and bruteforcing the problem by having, say 10k proxies each retrieve the state of 10k workers every second... may not be obviously impossible? Somewhat similar to sending/serving 10k DNS requests/s/worker? That's not trivial, but maybe not _that_ hard? (You've been working on modern Linux servers a lot more than I, but I'm thinking of e.g. https://blog.cloudflare.com/how-to-receive-a-million-packets...)
I did notice the sentence about "saturating our uplinks", but... assuming 1KB=8Kb of compressed critical state per worker, you'd end up with a peak bandwidth demand of about 80 Mbps of data per worker / per proxy; that may not be obviously impossible? (One could reduce _average_ bandwidth a lot by having the proxies mostly send some kind of "send changes since <...>" or "send all data unless its hash is <...>" query.)
(Obviously, bruteforcing the routing table does not get you out of doing _something_ more clever than that to tell the proxies about new workers joining/leaving the pool, and probably a hundred other tasks that I'm missing; but, as you imply, not all tasks are equally timing-critical.)
The other question I had while reading was why you need one failure/replication domain (originally, one global; soon, one per-region); if you shard worker state over 100 gossip (SWIM Corrosion) instances, obviously your proxies do need to join every sharded instance to build the global routing table - but bugs in replication per se should only take down 1/100th of your fleet, which would hit fewer customers (and, depending on the exact bug, may mean that customers with some redundancy and/or autoscaling stay up.) This wouldn't have helped in your exact case - perfectly replicating something that takes down your proxies - but might make a crash-stop of your consensus-ish protocol more tolerable?
Both of the questions above might lead to a less convenient programming model, which be enough reason on its own to scupper it; an article isn't necessarily improved by discussing every possible alternative; and again, I'm sure you guys have thought about this a lot more than I did (and/or that I got a couple of things embarassingly wrong). But, well, if you happen to be willing to entertain my questions I would appreciate it!
You're commenting on an article about people consuming less than $3/day; Americans on food stamps (SNAP appears to be about $4-6/day alone, not counting any other benefits) are a distraction, simply not part of the population that the article is discussing.
Picking anything but PlantUML's default theme will make PlantUML look less 90's-Java. As a minimalistic solution, I already like "skinparam monochrome true" https://plantuml.com/skinparam#:~:text=Black%20and%20White. (You can go further with CSS and themes and...)
I really like this article. I do think it's useful to consider that the unit of isolation ("process") of the cloud era is a VM or container, and that the major clouds do have some sort of permissions model.
The error rate is given per bit, not per second, i.e. every few bars represents a distinct DRAM chip. That makes some sense, and the article explains quite well why DRAM would behave like that... but I agree that I had to read the article at least twice to figure out that the x-axis on the graph represents the lower bit of the address line!
It's not ideal, but it's not a cryptographic risk.
Using an encoding that (like Protobuf) has multiple representations for a message may cause you problems if you switch implementations - sha256(encode(msg)) might yield different hashes on different implementations of encode().
But the main risk is an encoding that has multiple interpretations of a single encoding (e.g. sha256(encode("admin", "true")) == sha256(encode("admint", "rue"))), and Protobuf (can be unserialized, and thus) doesn't have that problem.
I was also thinking of promise pipelining, but note that the article proposes communicating A -> {B, C} -> D, never directly communicating from A to D. Cap 'n Proto "level 3" nodes could send promises from B and C to D, but that still needs A to talk to D, i.e. A -> {B, C, D}; {B, C} -> D. Same latency / depth of dependency chain, but still more messages - right?
(In return, note that Cap 'n Proto's A -> D message makes it more obvious how A figures out whether the operation succeeded; I'm not quite sure how that works in the proposed diagram. I suppose the proposed system actually puts all messages in a system-wide database, which does solve the problem.)
One pitfall to consider: software development is often its own culture, and nourishing and valuing (or merging!) multiple distinct cultures in a company requires real effort from leadership/management/team leads/senior ICs/...
I'm sure that e.g. truck drivers and software developers can be made into a strong team, but don't just assume that cross-culture collaboration will work "by accident"/"automatically".
The GCM (or XSalsa, or...) nonce is not secret - in almost all protocols the nonce goes over the wire in plaintext! - but must be a number-used-once. Those are quite distinct requirements.
All other things aside, if a bunch of articles get written about the phone, a lot of people think of Heineken ;-). I'm far from a marketing expert, but I have definitely seen less effective marketing campaigns.
It's partly a cost saving exercise, but also: running "chroot /var/empty /some/shitty/code" or putting "chroot /var/empty /some/shitty/code" in inetd.conf is useful. (On today's super-fast machines,) Firecracker starts fast enough to support such interactive uses, while giving you the extra security of a VM (i.e. greatly restricts what parts of the kernel and/or localhost the shitty code can talk to).
Yes! After a bit of playing around, I can follow Rust reasonably well... but only with an IDE, which I've never used for C or even really for the bits of Java I wrote. I understand that many more experienced Rust developers are similarly IDE-reliant.
I do think it's a deliberate tradeoff, having e.g. .push() do something useful for quite a few similar (Vec-like) data structures means you can often refactor Rust code to a similar data structure by changing one line... but it certainly doesn't make things as grep-friendly as C.
RSA-1024 is also a rather aggressive / dangerous choice, even before a quantum computer arrives: NIST SP 800-131A rev2 Table 2 and 5 forbid use of anything below 2048-bit RSA. Academic teams have actually broken up to RSA-829: https://en.m.wikipedia.org/wiki/RSA_Factoring_Challenge
Don’t roll your own crypto - just select a TLS cipher suite using Mozilla’s recommendations - but for hobby purposes, prefer Curve25519 or NIST P-256. Or play around with Kyber, post-quantum crypto is neat.