Airbnb Tech Talk: Evan Priestley of Facebook - Move Fast and Break Things(airbnb.com)
airbnb.com
Airbnb Tech Talk: Evan Priestley of Facebook - Move Fast and Break Things
https://www.airbnb.com/meetups/25z4mzn4a-tech-talk-evan-priestley
Move Fast & Break Things
A practical set of step-by-step recipes for reproducing some of Facebook's biggest mistakes (in security, web stacks, polytheism, deployment, algorithms, photo faxing and other areas) delivered by an engineer who helped make them.<p>About Evan
Evan Priestley worked as an infrastructure engineer at Facebook. He inflicted substantial and lasting harm, amassing a mountain of technical debt which the company will be paying down for decades to come.<p>Livestream
If you are not able to join us in person, we'll be streaming Evan's talk at 7pm. Please visit the meetup page to get the link for the livestream. A recording will be posted to our Tech Talk page as well.<p>Dinner, prepared by Airbnb's Executive Chef, will be served at 6pm.
6 comments
If an engineer at Bank of America said something to this tune, it'd have tremendous negative consequence as it should. Facebook is reaching a point where breaking things should not be okay simply because it opens up the possibility of breaking privacy settings and causing great damage.
Lots of people have pretty sensitive data within facebook with very granular privacy settings. When I read this attitude from facebook, it scares the shit out of me.
I'd love to hear more about why I shouldn't fear that their "Show this post to only x group of people" feature will break and result in everyone being able to see all my posts. What are your testing procedures for that and at what point are those kind of bugs caught?
Lots of people have pretty sensitive data within facebook with very granular privacy settings. When I read this attitude from facebook, it scares the shit out of me.
I'd love to hear more about why I shouldn't fear that their "Show this post to only x group of people" feature will break and result in everyone being able to see all my posts. What are your testing procedures for that and at what point are those kind of bugs caught?
Facebook is, by a huge margin, the buggiest release software that I use every day. When I look hard, I notice at least a dozen UI bugs a week. Yet, Facebook is hugely successful -- so successful, in fact, that it's hard for me to think of ways in which it could be more successful. Your comment implies that there is some threshold that they will pass, at which point they will be forced to adopt more stable sensibilities. There are 1 billion active users [1], so I ask: if they haven't gotten to this point, when exactly will they?
I don't believe the lack of quality in Facebook code poses an existential threat to Facebook, and I don't believe it ever will:
(1) Facebook code is not Google code, but it is not bad code, it's just not needlessly good code. If Google is consistently buggy, anyone can switch to Bing. On the other hand, switching from Facebook is really hard. Their core services are not easily replaceable. People are willing to put up with more. Note that this is even true in the case of businesses: Facebook is horrible to develop against, but developers don't have a choice.
As the network grows, this cost will be higher, which means Facebook can throw its weight around more, not less. This is exactly opposite of the situation you predict. The only thing that can really reverse this is if there is a viable competitor, but the increased growth of Facebook makes this increasingly unlikely. It is clear, in any event, that G+ is not quite there.
(2) By maintaining a lower bar, Facebook actually has an advantage. It is a website, and can fix its product at will. Code can be deployed quickly and rolled back accordingly, making failure cheap. This means they can concentrate on things that really matter like user acquisition.
(3) Facebook does not run nuclear reactors or bank systems. There is a high incentive to switch from buggy bank systems. There is little incentive to switch from buggy social network websites.
footnotes:
[1] whatever that means
I don't believe the lack of quality in Facebook code poses an existential threat to Facebook, and I don't believe it ever will:
(1) Facebook code is not Google code, but it is not bad code, it's just not needlessly good code. If Google is consistently buggy, anyone can switch to Bing. On the other hand, switching from Facebook is really hard. Their core services are not easily replaceable. People are willing to put up with more. Note that this is even true in the case of businesses: Facebook is horrible to develop against, but developers don't have a choice.
As the network grows, this cost will be higher, which means Facebook can throw its weight around more, not less. This is exactly opposite of the situation you predict. The only thing that can really reverse this is if there is a viable competitor, but the increased growth of Facebook makes this increasingly unlikely. It is clear, in any event, that G+ is not quite there.
(2) By maintaining a lower bar, Facebook actually has an advantage. It is a website, and can fix its product at will. Code can be deployed quickly and rolled back accordingly, making failure cheap. This means they can concentrate on things that really matter like user acquisition.
(3) Facebook does not run nuclear reactors or bank systems. There is a high incentive to switch from buggy bank systems. There is little incentive to switch from buggy social network websites.
footnotes:
[1] whatever that means
(3) Facebook does not run nuclear reactors or bank systems. There is a high incentive to switch from buggy bank systems. There is little incentive to switch from buggy social network websites.
I'm arguing that facebook wants to run something even more important than a banking system. For example, Facebook wants me to share posts that are only visible to me. Such a post could contain secrets that could have huge consequences on my life and life of others' if they became public. It is pretty clear from facebook's history that each year, they want you to share more of such private data.
They started with you being able to share your drunk party pics. They are headed towards actually becoming something like PayPal. Really, facebook should behave more like a bank than just-another-social-network if it wants its users remain confident. In this respect, looking at their past success to as proof that all is well is a mistake because I am certain facebook intends to be much more than what it is five years from now.
I'm arguing that facebook wants to run something even more important than a banking system. For example, Facebook wants me to share posts that are only visible to me. Such a post could contain secrets that could have huge consequences on my life and life of others' if they became public. It is pretty clear from facebook's history that each year, they want you to share more of such private data.
They started with you being able to share your drunk party pics. They are headed towards actually becoming something like PayPal. Really, facebook should behave more like a bank than just-another-social-network if it wants its users remain confident. In this respect, looking at their past success to as proof that all is well is a mistake because I am certain facebook intends to be much more than what it is five years from now.
The "break fast and move things" mantra isn't black and white, nor is it universal. It is a general alignment that, at the high level says, don't be afraid to make calculated mistakes but be prepared to quickly respond to feedback and those mistakes as appropriate.
That means a privacy or a security issue will be tackled with the utmost urgency--we would even shut down the site if we didn't have a quick fix for a security/privacy bug. However, if one of the Timeline aggregations isn't working or if a particular text box is not aligned, then we will fix it with the next release (usually in another day or half).
So it's not true that "move fast and break things" means "it's OK to introduce privacy/security holes" or even "it's OK to release shitty products". It just means we push early and push often.
That means a privacy or a security issue will be tackled with the utmost urgency--we would even shut down the site if we didn't have a quick fix for a security/privacy bug. However, if one of the Timeline aggregations isn't working or if a particular text box is not aligned, then we will fix it with the next release (usually in another day or half).
So it's not true that "move fast and break things" means "it's OK to introduce privacy/security holes" or even "it's OK to release shitty products". It just means we push early and push often.
Yes, you are not misunderstood, there is no need to clarify/interpret this slogan for us. The OP is saying that some mistakes are irreversible. Say, for example, a buggy privacy feature which -- because of fast and loose implementation -- allows users to see previously very private nude pics of a celebrity which he/she had only sent to their partner.
Even if it took only 30 seconds to find the error and the entire site was unplugged from the web so you could fix the bug, the damage is done, it's permanent and it's devastating. Facebook is crossing the line from failures being trivial to failures being potentially catastrophic. Sad, perhaps, by the standards around here, but some people conduct their lives on Facebook -- encouraged by Facebook to do so, mind you -- and you guys are treating it a little more seriously than a hackathon, from the sounds of it.
Even if it took only 30 seconds to find the error and the entire site was unplugged from the web so you could fix the bug, the damage is done, it's permanent and it's devastating. Facebook is crossing the line from failures being trivial to failures being potentially catastrophic. Sad, perhaps, by the standards around here, but some people conduct their lives on Facebook -- encouraged by Facebook to do so, mind you -- and you guys are treating it a little more seriously than a hackathon, from the sounds of it.
I feel that this kind of criticism is out of place, since Facebook has been working this way (successfully) for almost a decade without anything like that happening on a great scale.
Yes, mistakes can happen and they do happen everywhere, but Facebooks standards seem quite reasonable to me and they must be, since such mistakes would not be tolerated by its users.
Anyways, I doubt that Facebook ever encouraged its users to upload nude pictures. Whoever does this must be out of his/her mind.
Yes, mistakes can happen and they do happen everywhere, but Facebooks standards seem quite reasonable to me and they must be, since such mistakes would not be tolerated by its users.
Anyways, I doubt that Facebook ever encouraged its users to upload nude pictures. Whoever does this must be out of his/her mind.
While it's evident that Bank of America and Facebook are two different animals with entirely unique cultures, it's is worthwhile to mention that Facebook does balance the "Most fast, break things" mantra with proper control mechanisms.
For example, there is a layer called Gatekeeper that ensures only certain features are shown to users and that with a flick of a switch, code can be decommissioned in the event of a crash or breach.
For example, there is a layer called Gatekeeper that ensures only certain features are shown to users and that with a flick of a switch, code can be decommissioned in the event of a crash or breach.
Hell yes. Cases in point:
- Accidentally opening all profiles up, causing users to have to reset them.
- Accidentally setting all photos to public, causing users to have to reset them.
- Syncing many peoples contacts to @facebook email addresses, in many cases deleting the users original contact lists
- (the last straw) - letting private messages go onto the public wall. Facebook deny's this happened, but I personally know two people (no friend of a friend business) who this has happened to. In both cases there were very private messages from their ex-partners, the kind of thing they'd take extreme alarm to if they were on their wall. Both quickly deleted all messages when it happened, rather than saving the evidence.
Move fast and break things when there's private info at stake, is not the correct way to do things. I stopped posting to facebook and am going to close my account (and move to G+, who have better quality controls), and have non-tech friends who have decided that's the way to go after the private messages thing.
- Accidentally opening all profiles up, causing users to have to reset them.
- Accidentally setting all photos to public, causing users to have to reset them.
- Syncing many peoples contacts to @facebook email addresses, in many cases deleting the users original contact lists
- (the last straw) - letting private messages go onto the public wall. Facebook deny's this happened, but I personally know two people (no friend of a friend business) who this has happened to. In both cases there were very private messages from their ex-partners, the kind of thing they'd take extreme alarm to if they were on their wall. Both quickly deleted all messages when it happened, rather than saving the evidence.
Move fast and break things when there's private info at stake, is not the correct way to do things. I stopped posting to facebook and am going to close my account (and move to G+, who have better quality controls), and have non-tech friends who have decided that's the way to go after the private messages thing.
Evan is hilarious, check out all the copy on http://phabricator.org/ for an example.
However he should probably have been described as ex-Facebook, he left quite a while ago.
However he should probably have been described as ex-Facebook, he left quite a while ago.
This made me crack up the first time I read it!
'Facebook engineers rave about Phabricator, describing it with glowing terms like "okay" and "mandatory".'
'Facebook engineers rave about Phabricator, describing it with glowing terms like "okay" and "mandatory".'
This made me crack up even more : "Written in PHP, so literally anyone can contribute, even if they have no idea how to program."
Evan is the man - he is without doubt one of the funniest people I knew at Facebook.
We're also big fans of Phabricator over here at MemSQL.
We're also big fans of Phabricator over here at MemSQL.
Thanks for the link. I may use Phabricator just so I have an excuse to dig through the rest of the docs. :) Well, that and it looks like it could be quite useful.
His flavor text is also a great read:
http://www.phabricator.com/docs/phabricator/#flavortext
http://www.phabricator.com/docs/phabricator/#flavortext
This is such a grey are that I think it's close to impossible to make a rule about how much quality you should sacrifice for the sake of speed. Sometimes you might get lucky and break something insignificant, sometimes you create the IceBox Pro fiasco and be remembered at "that company". It's all calculated risk / gambling.
On the other hand, perhaps someone will argue that all publicity is good publicity.
On the other hand, perhaps someone will argue that all publicity is good publicity.
I honestly think that "Move fast and break things" might be the singularly most important software engineering mantra of our age.
Things break all the time in any company, but by understanding that frequent breakages occur, and adapting your development process to that understanding, you enable fast recovery and ultimately less breakage and greater security in the long run.
Things break all the time in any company, but by understanding that frequent breakages occur, and adapting your development process to that understanding, you enable fast recovery and ultimately less breakage and greater security in the long run.
This is inane, frankly. There are people who code to a much higher standard than this. People live and die by the code they write, so they can't accept "things break all the time ..." and forget about writing correct code.
See for example: your car, your bank (not the website, the backend), the space shuttle, the Fed, anything important. Some places cannot accept the supposed fact of frequent outages and failure, and engineer for that reality instead.
I don't know how you're concluding that the hackday coding methodology creates more secure code, unless you are simply unaware of the industrial standards to which other fields' programmers are held.
See for example: your car, your bank (not the website, the backend), the space shuttle, the Fed, anything important. Some places cannot accept the supposed fact of frequent outages and failure, and engineer for that reality instead.
I don't know how you're concluding that the hackday coding methodology creates more secure code, unless you are simply unaware of the industrial standards to which other fields' programmers are held.
Of course I wasn't talking about banks and space shuttles, that would be inane. Some places can't accept outages and failure, and naturally "move fast and break things" is an unacceptable mantra for those places.
However, few companies are rated on the quality of their code. Rather, most companies are rated on how well they serve their customers needs, including the vast majority of companies discussed on Hacker News.
The way to delight your customers is to iterate quickly. The way to achieve great software is also to iterate quickly. That is not a coincidence. Willingness to withstand a little bit of breakage allows companies to move fast. This contributes to an overall _higher_ level of quality, leading not to frequent outages (Facebook does not have frequent outages and failure), but to more robust product.
I am aware of how people build software in more traditional fields (my background is in compilers and static analysis). I am also aware that traditional software development techniques are being eclipsed by those who are willing to throw off the shackles of the past. The "hackday coding methodology" may not be useful to make the code for your car, but the techniques used to build your car are not useful for building a successful SaaS product.
However, few companies are rated on the quality of their code. Rather, most companies are rated on how well they serve their customers needs, including the vast majority of companies discussed on Hacker News.
The way to delight your customers is to iterate quickly. The way to achieve great software is also to iterate quickly. That is not a coincidence. Willingness to withstand a little bit of breakage allows companies to move fast. This contributes to an overall _higher_ level of quality, leading not to frequent outages (Facebook does not have frequent outages and failure), but to more robust product.
I am aware of how people build software in more traditional fields (my background is in compilers and static analysis). I am also aware that traditional software development techniques are being eclipsed by those who are willing to throw off the shackles of the past. The "hackday coding methodology" may not be useful to make the code for your car, but the techniques used to build your car are not useful for building a successful SaaS product.
Is it still suitable to code loosely when people use your trivial SaaS as if it were a vital component of their life? And if you encourage people to do so, isn't it irresponsible to keep applying such a methodology?
I think you're missing my point. It's not coding loosely, it's optimizing moving quickly. And by moving quickly you can actually have a safer and more secure site than by, say, only releasing code every 3 months or something.
This confuses the act of shipping code frequently with what the code actually does. Both of those are important but don't really have any bearing on each other.
fwiw, I disagree strongly that moving fast and breaking things "might be the singularly most important software engineering mantra of our age." Frankly, this terrifies me. You already qualify it by carving out banks, space shuttles (and presumably medical devices, transportation, etc). I suspect we could discuss a list of things where it doesn't quite apply and in the end I would have to add "...for products that aren't critical to your users." to the end of your sentence.
fwiw, I disagree strongly that moving fast and breaking things "might be the singularly most important software engineering mantra of our age." Frankly, this terrifies me. You already qualify it by carving out banks, space shuttles (and presumably medical devices, transportation, etc). I suspect we could discuss a list of things where it doesn't quite apply and in the end I would have to add "...for products that aren't critical to your users." to the end of your sentence.
"Move fast and break things" is about moving fast, and only mentions "break things" to indicate that its OK to break things. I agree that what the code does is orthogonal.
I would carve out space shuttles, etc, as different, but they already are carved out: they use special subsets of C with no memory allocation, run extensive static analysis and proof software, because those things literally cannot go wrong.
Anything that doesn't use that sort of thing has already committed to the idea that it won't be foolproof. If you code in a scripting language, you are already writing software which you implicitly acknowledge can go wrong. Software has bugs!
I would carve out space shuttles, etc, as different, but they already are carved out: they use special subsets of C with no memory allocation, run extensive static analysis and proof software, because those things literally cannot go wrong.
Anything that doesn't use that sort of thing has already committed to the idea that it won't be foolproof. If you code in a scripting language, you are already writing software which you implicitly acknowledge can go wrong. Software has bugs!
I guess my issue is specifically with 'break things' as it overgeneralises the idea of experimentation (which is something I do agree with). The impression that 'move fast, break things' leaves me with is that it's ok to break anything and that it's also (implicitly) not a big deal if they stay broken or somehow hurt a user (e.g by exposing, even temporarily, what was once private).
Even though it's a wonderfully pithy 4 words, people can interpret them in many different ways, most of which I'd argue are not good for the end-user.
"... you are already writing software which you implicitly acknowledge can go wrong. Software has bugs!"
Agreed, but shouldn't the intent be to create things in a way that tries to reduce the likelihood of damage while still allowing plenty of room for experimentation? i.e take some care that what you're about to do/push isn't going to do harm? If I follow the 'break things' mantra, I don't need to care. That's what bothers me. It's so easy to hide behind when something goes wrong.
Even though it's a wonderfully pithy 4 words, people can interpret them in many different ways, most of which I'd argue are not good for the end-user.
"... you are already writing software which you implicitly acknowledge can go wrong. Software has bugs!"
Agreed, but shouldn't the intent be to create things in a way that tries to reduce the likelihood of damage while still allowing plenty of room for experimentation? i.e take some care that what you're about to do/push isn't going to do harm? If I follow the 'break things' mantra, I don't need to care. That's what bothers me. It's so easy to hide behind when something goes wrong.
From what I've learnt of Facebook's attitude, it's more that you won't be fired if you take the site down (at least, the first time). I don't believe that they don't care when you break things, and that wasn't what I was advocating.
Yup, Facebook's broken for me right now.
I wish I could know, whether he tried moving slow and make things less brittle.