That article on rapid Rust prototyping matches my experience with using Rust as the backend for a web and iOS app. I used clone() and Vec and String and other shortcuts from that article as much as possible since I was building a backend application versus an operating system. It enabled a lot more velocity and made it fun to add features. And it was still blazing fast.
If anyone is considering using Rust and is nervous about lifetimes and bare metal, check out that article and try its guidance. I learned these things on my own the hard way and would have loved to read this article 18 months ago. It's really quite good.
It was actually Rainbow's End by Vernor Vinge that really got Zuck excited about the possibilities of VR—he read it a few years before he bought Oculus and talked about it a lot at the time.
I was at FB from 2006-2015, and Scuba caused a phase change in how Facebook's performance was analyzed and how quickly insights were gained. For many of us in Infrastructure, we spoke of the pre-Scuba, post-Scuba eras. It was such a joy to have aggregated realtime performance metrics for our internet scale services and have the ability to drill down on just about any dimension when doing perf investigations. It gave us so much more confidence in our systems, both in our ability to detect issues, as well as pinpoint what was causing them.
I'd also like to echo what spicyj said about the UI, it is very usable. It was common for managers, PMs, and even people in completely non-technical business roles to use Scuba. It was my favorite internal product at FB and the one I miss the most.
Cartographer ingests BGP topology and POP and datacenter health (and other things), and then pushes what is basicially a lookup table to tinydns. None of the dynamism is inside the DNS server itself, which lets it focus on what it is good at--responding to a crap ton of DNS requests per second.
Cost was a smaller factor than politics; the Indian government wanted the private keys for our certs in order to let FB put a POP there. That was an absolute dealbreaker, so we served India from Singapore and other POPs in nearby countries.
Regarding building a datacenter, that's much higher risk since it's a $100M-1B capital investment. I'm guessing both Facebook and Google see too much volatility to make the risk worth it. It could be an expensive paperweight if the Indian government changed their minds.
I joined FB in 2006 and had never been in an open office environment before. It was quite an adjustment. I didn't like working in the office until I had been there for about six months. The office on University Ave was packed tight and didn't have a lot of creature comforts--it was cheap desks arranged in groups of four, some conference rooms, a break room with snacks, and that's it. And I had come from a cubicle farm at Apple.
There were definitely large benefits to the open floor plan back then since we were working on so many things and each person was wearing so many hats. It made staying in touch with people's progress really easy. We were also working 80-100 hours a week back then, so it was good to constantly stay in touch since there was so little structure and process back then.
I have mixed feelings about FB being held up as one of the models for open floor plans today though. While I think it was definitely useful at the time, I think the open floor plan has become less important as tools have gotten better. We didn't have tools like FB Groups, Phabricator, etc, so you had to be within earshot of your collaborators in order to make sure everyone was in sync.
My sweet spot would be team sized rooms if I was starting a company today. I used to be so jealous of teams that had to grab war rooms and ensconce themselves in there with a bunch of laptops and displays. Even though they were working super hard to meet deadlines, they looked happy as can be.
Indeed, but there is something to be said for sane defaults--look at the thread on HN a few weeks ago about SSH cipher defaults. Yes, TLS and the CA ecosystem has its faults, but my stance is that any chat protocol should be using auth and encryption, especially group chat protocols like IRC. I'm less upset now after seeing that the proposed spec for IRC 3.3 includes something similar to HSTS.
I read IRC v3 and assumed they were making major changes to the wire by bumping the major version, which is why I said what I said. I've been using IRC for 24 years but haven't implemented it before, maybe I'll do that now so I can have a better idea of what I'm talking about.
Hmm, I didn't see the 3.3 spec which is still being developed, it looks like they are doing strict transport security as a base extension. So that would address most of my concerns I think. I would still prefer to see TLS required no matter what.
Fair enough, but I feel you're being a little pedantic since there are base extensions described which are required, and optional extensions which are not required. TLS is one of the optional extensions.
So I'll reframe my question and ask: Why isn't TLS a base extension?
Why isn't encryption baked in by default? It's optional from what I can tell. I don't know how you can design a new protocol and not include encryption.
I think you're being overly harsh. I read the entire article and enjoyed it very much, including the author's commentary.
I didn't interpret their tone as trying to be hard-edged or snarky; I felt it was just them honestly showing the process of someone returning home to a place that was incredibly isolated 25 years ago, and then trying to rectify what it's like for these kids in that same town who now have access to so much more information and avenues of communication.
I liked the different asides where they mentioned what was popular in the town 25 years ago versus what the kids like now, and how the kids interests today compare to bigger city tastes. I grew up in a town that's 5x larger than Napoleon but still pretty small, and I see many parallels with this story that match my experiences.
Not sure if you know this, but you can ask Google to blur a Street View address by filling out a form. I'd guess he did this himself versus any sort of conspiracy thing happening.
To your last paragraph, only relying on forgetting the keys works great, as long as you have absolute 100% confidence in the mechanism used to do that. I read your posts on HN often so I know you know you're quite familiar with defense in depth--I feel that user data is one of those areas where it's ok to do more than one thing to protect the data.
That said, there are a number of systems at FB where deleting a crypto key loses the linked data forever--but they still crunch the hard drives just to be really sure. The drive crunching is an incredibly tiny expenditure compared to the massive CapEx and OpEx required to build, stock, and run the datacenters. It's worth it if only for the peace of mind.
Lucid Emacs -> Xemacs was mentioned in a sibling comment, but I remember people being in awe about EB working on those. There were many interesting employees back then--J. Chenault, J. Gehlen, R. Braumeller, F. Eiden, K. Sears, etc. I've lost touch with most of them but I learned a lot from them.
I started in October 1997 and did Perl work on backend systems, but I had a few friends who worked on the frontend. They used to talk about how Catsubst had a very small set of verbs and that everything was a function, even arithmetic, and that prefix notation was used for all macro calls. They thought it was Lisp-inspired, but they were also pretty junior programmers too. I never worked with Catsubst myself, so my information is second-hand, while yours is zero-hand since you wrote it. :-)
Thanks for all the work you did on Amazon's infra!
There was more Lisp influence than just the CS macros you mention here. Eric Benson was a major contributor to Lucid Emacs and many early Amazon engineers had Lisp experience. SICP was practically issued to every new software engineer--it was that popular.
The template language for Amazon's website at the time (catsubst) was a Lisp-inspired prefix notation language with a relatively small number of functions. For example, there was an add macro but not a subtract macro, so to do A-B, you had to say ${add A -B}. I might have the exact syntax wrong since it's been 20 years.
The two AsiaBSD papers linked from the post are good for more detail. I was a little surprised they had to hack sendfile to do the crypto in the kernel in order to get the throughput they're used to with http, but the reasons are explained in the papers.
However, I'm quite surprised Netflix went with Intel's ISA-L library for AES-GCM given Intel's perf gains were so very marginal compared to BoringSSL. I would have gone with the library that had more eyeballs on it, and in general I'd give Google the edge over writing solid, secure code than I would Intel.