Twitter SVP Chris Fry Breaks Down How His Engineering Org Works(recode.net)
recode.net
Twitter SVP Chris Fry Breaks Down How His Engineering Org Works
http://recode.net/2014/01/02/twitter-svp-chris-fry-breaks-down-how-his-engineering-org-works/
9 comments
He has no eyebrows.
Chris mentions the 3 things that motivate people. Good video about this from RSA Animate if you haven't seen it yet:
http://www.youtube.com/watch?v=u6XAPnuFjJc
http://www.youtube.com/watch?v=u6XAPnuFjJc
I've been trying to use Dan Pink's ideas on motivation for a while now, it was good to hear someone at his level mention them. Hadn't watched the video in a while, thanks for linking it.
While Twitter a is cool service I can't imagine what 1000 engineers can do all day long? Based on this maybe they are building their organization wrongly and following their footsteps isn't the correct way to follow.
This is pretty common in the growth phase of a company, especially one with easily scalable demand such as a software company.
As others have pointed out, a lot of projects have been built from 'scratch' due to Twitter's needs and/or culture. This alone has a heavy carrying cost, as they now have the burden of having at least two maintainers for mission critical applications and services. It doesn't mean the engineer-tuples are bijective onto the various projects, but there are only so many tickets a person can resolve.
At Amazon, we grew to what we then thought was an insane size in IT (and across the company – we had horrid little offices all over the place in Seattle). This was boosted by a number of acquisitions, such as drugstore.com and homegrocer.com. In 2000, we had a small round of layoffs, then they had another round in 2001.
I would expect something similar at Twitter if they don't move horizontally. I would say that, given the competitive HR landscape, it's quite possible they intentionally overhired to secure strong engineers and will rank and yank those who have not turned out to be performers.
They're still in an annealing phase, but, honestly, I'm unqualified to speculate on their internal alignment, growth plans, or any RIFs they have planned in the near future.
As others have pointed out, a lot of projects have been built from 'scratch' due to Twitter's needs and/or culture. This alone has a heavy carrying cost, as they now have the burden of having at least two maintainers for mission critical applications and services. It doesn't mean the engineer-tuples are bijective onto the various projects, but there are only so many tickets a person can resolve.
At Amazon, we grew to what we then thought was an insane size in IT (and across the company – we had horrid little offices all over the place in Seattle). This was boosted by a number of acquisitions, such as drugstore.com and homegrocer.com. In 2000, we had a small round of layoffs, then they had another round in 2001.
I would expect something similar at Twitter if they don't move horizontally. I would say that, given the competitive HR landscape, it's quite possible they intentionally overhired to secure strong engineers and will rank and yank those who have not turned out to be performers.
They're still in an annealing phase, but, honestly, I'm unqualified to speculate on their internal alignment, growth plans, or any RIFs they have planned in the near future.
[deleted]
But think what they have the power to do.
If you've worked on online software you might have some appreciation of the masses of unseen services that are essential for running the system. All web apps, except in the early days, are icebergs. Payments, analytics, control panels, monitoring, deployment etc. -- there is just a huge amount of stuff that goes on that the user typically doesn't see.
I have actually built and scaled a system similar to Twitter called Plurk (which has millions of users, billions of data items and similar use cases as Twitter). We built this with a team of a few developers and one sys-admin. While Twitter is at a much larger scale, I really doubt you need 1000 engineers to solve that problem, because even at Plurk's scale we built automated systems for scaling, monitoring, backups etc. (e.g. sharded the databases, used master-master MySQL setups that supported auto-failovers, nagios for alerts etc.)
You may say: but they have mobile apps? But so do a lot of other companies that don't have 1000 engineers. You need a few really great mobile devs to build great mobile apps, again based on my experience.
I think what has happened for them is that their structure isn't that good and they have scaled too fast (and in this process they have hired a lot of B and C people that aren't that good). It's a least my conclusion based on the limited information I know about them.
You may say: but they have mobile apps? But so do a lot of other companies that don't have 1000 engineers. You need a few really great mobile devs to build great mobile apps, again based on my experience.
I think what has happened for them is that their structure isn't that good and they have scaled too fast (and in this process they have hired a lot of B and C people that aren't that good). It's a least my conclusion based on the limited information I know about them.
If Plurk has 5 developers serving 1 million daily users, and Twitter has 1000 developers serving 100 million daily active users, you only have to believe that scaling by 2 orders of magnitude is twice as hard for it to be a tie in terms of developer:user ratio.
(I suspect the write load is rather different - the figures I can find for Plurk suggest 8 billion messages in total - Twitter writes half a billion a day.)
(I suspect the write load is rather different - the figures I can find for Plurk suggest 8 billion messages in total - Twitter writes half a billion a day.)
I don't think there's a huge differences serving 1 million daily users or serving 100 million. In both cases you need highly scaleable systems. In 100 million case you need a lot more computers, but I think the overall structure of the systems could be very similar (e.g. sharded databases, auto fail-overs, extensive monitoring etc.) Of course you need a larger team serving 100million users, but I don't think you need to scale the team as much as Twitter has done.
The big advantage of software though is that you don't have to maintain a fixed ratio of developer:user like that, though. Sure, your employee count should increase as user count does, but far less than linearly.
I'm not sure this is in practice true. Twitter have to start writing or tuning their own scale-specific software, need more analytics, application security, developer productivity, partner and sales engineering etc. proportionally than they would do at 1% of their size. Certainly they could run 100 Plurks with many fewer than 1000 engineers but I'm not sure that's the best way to characterize Twitter.
He suggests that it might scale logarithmically, not linearly.
Take a look at https://github.com/twitter this should give some insight into where they may be spending their time.
There are a few projects that would be well known (Storm, Kestrel, Bootstrap, FlockDB). Although I'm 100% if they were created at Twitter or just hired in.
There are a few projects that would be well known (Storm, Kestrel, Bootstrap, FlockDB). Although I'm 100% if they were created at Twitter or just hired in.
I for one would be (honestly) interested in reading your article in the same vein as the original - what would you say is and is not necessary in his take?
one of the problems there is that everything is extremely convoluted.... every single thing is a huge rat's nest of cross team dependencies. I found it an extremely frustrating environment to try and work in.
Their monitoring dev team alone is 22 [1], and they specifically mention decomposition of services as you say.
[1] https://blog.twitter.com/2013/observability-at-twitter
[1] https://blog.twitter.com/2013/observability-at-twitter
Come on. We're supposed to doubt the merits of this entire article about Twitter's philosophy and general large-organization engineering practices because someone outside can't imagine how so many people fill their day? Because maybe a thing which we can't hope to gain an accurate perspective of indicates they're doing everything wrong?
This reminds me of the story of a walk from SF to LA illustrating why software estimates are always so wrong:
http://www.quora.com/Engineering-Management/Why-are-software...
Instead shouldn't we just evaluate the content on its own merits?
This reminds me of the story of a walk from SF to LA illustrating why software estimates are always so wrong:
http://www.quora.com/Engineering-Management/Why-are-software...
Instead shouldn't we just evaluate the content on its own merits?
Look at e.g. Instagram or imgur or reddit or HN for a lean team. It's hard to compare without intimate knowledge of what either team faces, but without considering alternatives, many things that shouldn't look plausible actually do, e.g. The $5,000 hammers the army supposedly bought, the (incorrect, but oft repeated) space pen story, or the clusterfuck that is US healthcare.
They probably make awesome unit test scripts.
This is a bit of an unfair request, but it would be interesting to have you break down the engineering groups you think exist at Twitter and estimate their size.
He highlights Twitter EventParrot and MagicRecs, both compelling experiments:
EventParrot for news alerts: http://www.theverge.com/2013/10/10/4823278/twitter-eventparr...
MagicRecs for notifications: http://www.theverge.com/2013/9/24/4767290/twitter-will-notif...
EventParrot for news alerts: http://www.theverge.com/2013/10/10/4823278/twitter-eventparr...
MagicRecs for notifications: http://www.theverge.com/2013/9/24/4767290/twitter-will-notif...
"Breaks Down How His Engineering Org Works" is an exaggeration. "Offers a glimpse" would have been more appropriate:
1000+ engineers that include (perhaps a majority) of people working on reliability/operations.
Mobility - engineers _can_ move every quarter to a team that has an open position
Peer-feedback (360?) promotions based system (this is rather popular)
I guess I was hoping for something more about the actual structure and interactions, but it's mostly saying it's not too centralized, nor too distributed and that it's "like a school" and they're using lean/agile methodologies.
1000+ engineers that include (perhaps a majority) of people working on reliability/operations.
Mobility - engineers _can_ move every quarter to a team that has an open position
Peer-feedback (360?) promotions based system (this is rather popular)
I guess I was hoping for something more about the actual structure and interactions, but it's mostly saying it's not too centralized, nor too distributed and that it's "like a school" and they're using lean/agile methodologies.
I like their counterpoint to the idea that you have to dump all the managers. It really is about managerial effectiveness.
I also like their idea of a free market for transfers within the company. This is a big departure from many large companies where your boss has to sign off on your move. Ultimately that creates an environment where you need a job offer in hand to switch.
I also like their idea of a free market for transfers within the company. This is a big departure from many large companies where your boss has to sign off on your move. Ultimately that creates an environment where you need a job offer in hand to switch.
Well... it looks like there needs to be an opening first (which is the obvious thing, I guess) and also who decides when there is a single place but 4 applicants? It sounds like the team would decide. After what? interviews?
That's usually how it works for internal transfers. If the person is well known enough, then the team (or manager) may decide without as formal an interview process.
Regarding the dig at stack ranking: it's really just Yahoo now. Microsoft abandoned it 2 months ago. [1]
[1] http://www.theverge.com/2013/11/12/5094864/microsoft-kills-s...
[1] http://www.theverge.com/2013/11/12/5094864/microsoft-kills-s...
Every company is stack ranking and every good manager (and many bad ones) have their teams ranked in the event that they need to cut or have the opportunity to improve the team through training, promotion, etc. ("What would you do if I gave you four open positions?")
You have that guy who you're putting on plan because he's late with his deliverables, doesn't really 'get' the purpose of the department/group/company, and kvetches all the time about one thing or another? You've ranked him.
That woman you're trying to mentor as much as possible so you can promote her, maybe even to your peer level? Ranked.
There are people who are essential to an organization at different times. Keeping the lights on? I've cut everything but devops and maintenance, including myself. Sometimes you can save someone, but you can't transfer a problem.
You have that guy who you're putting on plan because he's late with his deliverables, doesn't really 'get' the purpose of the department/group/company, and kvetches all the time about one thing or another? You've ranked him.
That woman you're trying to mentor as much as possible so you can promote her, maybe even to your peer level? Ranked.
There are people who are essential to an organization at different times. Keeping the lights on? I've cut everything but devops and maintenance, including myself. Sometimes you can save someone, but you can't transfer a problem.
It might only be temporary at Yahoo. If they have a lot of C players, they might be calculating that there won't be a lot of false positives in the early purges.
eBay also has stack ranking.
1000 engineers is not a very lean organization. Maybe they sit around all day and try to figure out some way for Twitter to actually make money? Or maybe they count everyone tech related as an engineer, like designers and QA and build automation, etc? Google says Twitter has 2,300 employees.