Google's infamous internal 2010 “I just want to serve 5TB” video now public(youtube.com)
youtube.com
Google's infamous internal 2010 “I just want to serve 5TB” video now public
https://www.youtube.com/watch?v=3t6L-FlfeaI
513 comments
Background: https://rachelbythebay.com/w/2021/10/30/5tb/
This video was hugely influential on changing the way Google does internal tools and operations.
This video was hugely influential on changing the way Google does internal tools and operations.
As a Googler, it's often easier for me to setup a GCP consumer account, AWS, or Heroku account to demo something, compared to using anything internal. I remember the most annoying situation was like 10 years ago when me and other engineers ported Quake 2 to run in Chrome, we were in a time crunch to demo it multiplayer, and I ended up setting up an AWS account to serve it. But then I left it running and forgot and ended up getting a few hundred dollars billed to me because the Quake2 server was chewing CPU.
I wish I would have had this video before 2010. I got paged at night every time there was a PCR failover, and I didn’t know what to do with it. This video is better than all the extensive documentation that we had.
Ah, this takes me back (disclaimer: Xooger, 2010-2017). It's painful and funny because it's true. Or was true.
Rumour had it that the Borgmon readability requirement was removed when Sergey saw this video. I don't know if this is true but that's what I heard.
Rumour had it that the Borgmon readability requirement was removed when Sergey saw this video. I don't know if this is true but that's what I heard.
Ah, the laughs (Xoogler since 2020).
It was a lot easier, at least last year: you'd use "flex" quota from your PA pool (product area) for Spanner and Borg, write some code for your server, a few configs here and there, and you'd be ready and serving.
When I first started at Google I got things done a lot faster because I didn't know all those rules existed and nobody stopped me. My service was still plenty fast & reliable.
Eventually it all got rewritten by other people to do things properly like the video says.
The same conversation at Netflix 10 years ago:
I want to serve 5TB of data.
Ok, spin up an instance in AWS and put it there.
I want it production ready.
Ok, replicate it to a second instance. If it breaks we'll page you to fix it.
The funny thing is, for important stuff, we ended up doing similar things to what you see in this video, but for unimportant things, we didn't. I think it was a better system, and it was amusing when we hired people from Google who were confused by the lack of process and approvals.
I want to serve 5TB of data.
Ok, spin up an instance in AWS and put it there.
I want it production ready.
Ok, replicate it to a second instance. If it breaks we'll page you to fix it.
The funny thing is, for important stuff, we ended up doing similar things to what you see in this video, but for unimportant things, we didn't. I think it was a better system, and it was amusing when we hired people from Google who were confused by the lack of process and approvals.
Non-Googler: What do all those words mean?
Noogler: Haha, this video is so funny!
L4 SWE: (Crying because the video is so true)
L5 SWE: Haha, this video is so funny! I should show it to my interns, this will be a good training for them.
L6+ SWE: Why do people think this is funny? This Broccoli Man guy makes some really good points...
Noogler: Haha, this video is so funny!
L4 SWE: (Crying because the video is so true)
L5 SWE: Haha, this video is so funny! I should show it to my interns, this will be a good training for them.
L6+ SWE: Why do people think this is funny? This Broccoli Man guy makes some really good points...
At 2:05 the green dude asks if you think your users are scum and do you hate them.
The funny thing is Google as an org ends up hating their users "accidently" anyway because of their history of pulling the rug under the services/APIs etc.
The funny thing is Google as an org ends up hating their users "accidently" anyway because of their history of pulling the rug under the services/APIs etc.
Recent and related:
I don’t know how to count that low - https://news.ycombinator.com/item?id=28988281 - Oct 2021 (259 comments)
especially this comment: https://news.ycombinator.com/item?id=29032656
I don’t know how to count that low - https://news.ycombinator.com/item?id=28988281 - Oct 2021 (259 comments)
especially this comment: https://news.ycombinator.com/item?id=29032656
Hey... those of us that worked on Google's internal Bigtable service worked very hard so you didn't have a file to a ticket to set up replication between your Bigtable cells.
The rest does seem about accurate though.
The rest does seem about accurate though.
What exactly are these "peer bonuses"? Is it real? Is it what I think it is? Do people actually use them as bargain chips?
As an external user who has found Google's services to be incomprehensible, it's nice to know it is (was) equally as painful internally.
I’m so confused, isn’t this just like basic highly available infrastructure mixed with a toxic SRE culture?
I want to serve 5TB!
Okay grab two instances in different patching zones, create a bucket in our replicated RADOS storage that can hold your data or create a table/db in our Postgres cluster, write your app with tests, add an entry in to the load balancer, add an entry in our big ole distributed job scheduler if you need cron, and submit a PR against the infra repo to add Prometheus metrics and alerts.
And when your done with that set up CI/CD because you shouldn’t assume that instances are reliable and if you don’t give us the code to do a deploy we can’t recreate your app when the VM goes belly up and we’ll have to page you.
Are people not used to what it really takes to “just run some code?”
I want to serve 5TB!
Okay grab two instances in different patching zones, create a bucket in our replicated RADOS storage that can hold your data or create a table/db in our Postgres cluster, write your app with tests, add an entry in to the load balancer, add an entry in our big ole distributed job scheduler if you need cron, and submit a PR against the infra repo to add Prometheus metrics and alerts.
And when your done with that set up CI/CD because you shouldn’t assume that instances are reliable and if you don’t give us the code to do a deploy we can’t recreate your app when the VM goes belly up and we’ll have to page you.
Are people not used to what it really takes to “just run some code?”
Only if you think your users are scum. Do you think your users are scum? Do you? Why do you hate your users?
Holy crap! I've been asking for this for forever[1]. Thank you to the leaker!
[1] https://news.ycombinator.com/item?id=21786729
[1] https://news.ycombinator.com/item?id=21786729
As a friend of mine explained why she left Google a year or so ago, "I got tired of emailing 30 people to try to figure out who owned a single variable."
The borgman readability approvers makes me chuckle.
At Stripe, there were language approvers. Only those blessed could approve PRs. Even XML had a set of approvers. I had fun time getting hold of an XML approver.
At Stripe, there were language approvers. Only those blessed could approve PRs. Even XML had a set of approvers. I had fun time getting hold of an XML approver.
this is making me miss memegen
google had its downs, but wasting hours on memegen was not one of them
google had its downs, but wasting hours on memegen was not one of them
I didn't realize this was made by Google in the first place when I saw it a few days ago. I hope things are simpler now, but I doubt it.
what am I looking at here
EDIT: it has been explained to me: https://rachelbythebay.com/w/2021/10/30/5tb/
EDIT: it has been explained to me: https://rachelbythebay.com/w/2021/10/30/5tb/
There's almost some kind of irony on uploading that to youtube, a feeling of "why can't I deploy my service as easily as people can upload videos to youtube?".
The multiple repetitions of “This is Google” hit home for me. I never worked as a software engineer so much of the rest is out of scope to my experience, but the constant idolization of Google, and by proxy each other for working at such a place, eventually changed from feeling coy to cultish.
For anyone else who'd rather read than watch this video, here's the transcript (from YT's auto-generated captions): https://pastebin.com/8UrFftM6
Wow, way to trigger my Google PTSD. :-)
I think the only thing I would have added would be using some service that recently was changed so that someone could get promoted but does the same thing the old one did but with different bugs.
I think the only thing I would have added would be using some service that recently was changed so that someone could get promoted but does the same thing the old one did but with different bugs.
Facebook had its own meme: "Pusher I need a hotfix"
What does “serve 5TB” refer to? They expect 5TB of network bandwidth over some time period (a month?)? Or their database takes up 5TB on disk?
> It was a site where you could type in a script and it would do text-to-speech and actually
> animated some goofy characters to lip-sync the words for you
Does anyone still have the code for generating "Fox girl and Broccoli man" videos? I'm thinking of starting a small revival of this meme.
Does anyone still have the code for generating "Fox girl and Broccoli man" videos? I'm thinking of starting a small revival of this meme.
Wow, kind of funny that Xtranormal now lives on in the few viral videos that were made with it.
Here's where the company is now (the original domain is used for something else now):
https://en.wikipedia.org/wiki/Nawmal
Here's where the company is now (the original domain is used for something else now):
https://en.wikipedia.org/wiki/Nawmal
This video basically is making fun of a common situation of Google at the time, where a person wants to serve up some data for analytics, but the sysadmins expect the person to follow a process intended for much more complex and high availability services run by teams of skilled engineers.
It parodies SRE as a BOFH sysadmin, even though in general SRE are quite easygoing and helpful.
It helped poked fun at a number of overly stuffy processes and also helped push people to make hosting modest datasets (like this 5TB one) easier.