iOS at Facebook [pdf](static1.squarespace.com)
static1.squarespace.com
iOS at Facebook [pdf]
https://static1.squarespace.com/static/5463eca4e4b09c644a61f99a/t/55fa801ae4b008853e3de038/1442480154751/SimonWhitaker.pdf
23 comments
Yeah, the difference between Facebook and Instagram or Twitter is incremental. They basically do the same stuff, Facebook does a little bit more (although, they entirely removed direct messages). Why is the Facebook app so huge?
I wouldn't be surprised if all the messages code is still in the app.
I've got an alpha of the Facebook for Android app and it's got all of my conversations loaded, including text from the last message. It'll even light up with a red indicator when I get a new message. All the actions start an Intent which launches my Messenger app, though.
So yeah, it's probably all dead code that's still there.
So yeah, it's probably all dead code that's still there.
>It's all a bunch of bored engineers looking for ways to be clever but without anyone to take ownership of the codebase.
You couldn't think of a more charitable interpretation? I'm not one to care about rules but the hn guidelines on commenting are important for making sure the community doesn't eat itself alive like reddit did/does.
>The app itself doesn't do that much.
Right, but in the presentation he made it clear that it was less about what the app does and more about the number of engineers making changes to it. Given the size of their teams all making their own changes to the iOS codebase, doesn't it make sense to modularize it so they aren't tripping over each other breaking the application constantly? They don't have a dedicated iOS team and having one would bottleneck the other teams.
You couldn't think of a more charitable interpretation? I'm not one to care about rules but the hn guidelines on commenting are important for making sure the community doesn't eat itself alive like reddit did/does.
>The app itself doesn't do that much.
Right, but in the presentation he made it clear that it was less about what the app does and more about the number of engineers making changes to it. Given the size of their teams all making their own changes to the iOS codebase, doesn't it make sense to modularize it so they aren't tripping over each other breaking the application constantly? They don't have a dedicated iOS team and having one would bottleneck the other teams.
> Right, but in the presentation he made it clear that it was less about what the app does and more about the number of engineers making changes to it.
Well if it doesn't do that much, why are so many engineers working on it? Perhaps they have so many people working on it because they are all redoing the same stuff and not working together?
Well if it doesn't do that much, why are so many engineers working on it? Perhaps they have so many people working on it because they are all redoing the same stuff and not working together?
It's interesting to note that this presentation strongly contradicts the advice of "The Pragmatic Programmer" and "Clean Code", which are highly respected by many. Does this mean that those books are wrong, or does it mean that Facebook will eventually come to regret their philosophy? Or maybe writing this kind of code is somehow fundamentally different, so the usual wisdom and rules of thumb don't apply.
I really don't know the answer, but find it fascinating and can't wait to see what Facebook is doing in 10 years time.
I really don't know the answer, but find it fascinating and can't wait to see what Facebook is doing in 10 years time.
I think what it means is that as it stands now, there is no definite good/bad way to make successful software. There are just opinions.
There are definite bad ways. I've watched a few companies go down the shitter using known bad ways.
I've watched a few companies go down the shitter using known good ways; and seen companies with mind-blowingly horrific software succeed based on marketing and contracts. There are a lot of anecdotes, but I have yet to see any data suggesting a correlation.
It also depends on your definition of successful.
To be successful does software need to (check off what you think would matter):
* Make money directly
* Make money in-directly
* Be used by at least 1 user (not including dev + dev friends and family)
* Be alive and working in 6 months
* Be alive and working in 2 years
* Be alive and working in 10 years
* Simply ship so dev gets a positive yearly review
* Be bug free
* Only crash N% of the time (who cares just re-run it and don't do the bad thing)
* Quality / features don't matter as long as marketing is talking about it and people play with it so it looks successful.
* Simply exist, but be subsidized by some one or something so devs keep hacking on it even if no one uses it.
* Still be useful when there are zero devs working on it.
To be successful does software need to (check off what you think would matter):
* Make money directly
* Make money in-directly
* Be used by at least 1 user (not including dev + dev friends and family)
* Be alive and working in 6 months
* Be alive and working in 2 years
* Be alive and working in 10 years
* Simply ship so dev gets a positive yearly review
* Be bug free
* Only crash N% of the time (who cares just re-run it and don't do the bad thing)
* Quality / features don't matter as long as marketing is talking about it and people play with it so it looks successful.
* Simply exist, but be subsidized by some one or something so devs keep hacking on it even if no one uses it.
* Still be useful when there are zero devs working on it.
They probably have a while to go before they hit the ceiling, if Google's news last week about holding over 2 billion lines of code in one repository means anything.
SRC: http://www.wired.com/2015/09/google-2-billion-lines-codeand-...
SRC: http://www.wired.com/2015/09/google-2-billion-lines-codeand-...
This didn't really tell me why Facebook needs 400 people to work on the iOS app and 75 commits per hour.
Facebook doesn't have a dedicated iOS team, rather each product team (feed, photos, search, messages, etc.) commits code to the app. This allows updates to be made faster and more consistent since there aren't multiple product teams for each product for each platform, but it also leads to having a large number of contributors, commits and duplicate code.
What's crazy is that WhatApp has 4 Android engineers and 1 billion downloads.
I would suggest that WhatsApp does a whole lot less.
I agree, and I think that focus is actually a strength of the product.
> What's crazy is that WhatApp has 4 Android engineers
Interesting. Any source for this claim? And also what's the full size of Whatsapp team?
Interesting. Any source for this claim? And also what's the full size of Whatsapp team?
Jan Koum on Twitter when their app crossed 1 billion users on the Google Play Store[0]. Not sure what their current team size is, but it seems like it's around 40-50. Last year at Startup School, Jan said the backend team 7-8 people on the backend[1].
[0] - https://twitter.com/jankoum/status/575740639444336640
[1] - https://youtu.be/8-pJa11YvCs?t=1272
[0] - https://twitter.com/jankoum/status/575740639444336640
[1] - https://youtu.be/8-pJa11YvCs?t=1272
[deleted]
Surely most of these commits are either automated, or similar in nature (i.e., adding some static resources to some file).
> Here are the stats for the week starting 24 August this year.In that week: 429 unique contributors 2823 total commits
Right below it:
> Doesn’t include commits to bump version numbers, add translated strings etc - actual number is over 4,000
Any way you slice it that is a chaotic pace of change.
Right below it:
> Doesn’t include commits to bump version numbers, add translated strings etc - actual number is over 4,000
Any way you slice it that is a chaotic pace of change.
I doubt they have 400 people dedicated to the iOS app. I think it's more that 400 people happened to have touched the iOS app.
So the main purpose of most of the 18000 classes is to make it possible for the app to scale up to 18000 classes? :)
Partly! The tools they mentioned (IDE, Mercurial) don't impact the app itself though. Which makes them irrelevant and doesn't factor into why their codebase is massive.
They are relevant if you take them for what they are: examples that explains, in a general way, why FB has large codebase. Because when something doesn't fits their needs they will write their own version.
So I guess with iOS they wrote/rewrote some part of the system functionnalities to better fit their needs.
So I guess with iOS they wrote/rewrote some part of the system functionnalities to better fit their needs.
FYI when Sun owned the FB HQ building the address was 1 Network Drive :)
Can't even imagine the amount of crap in their code-base. I bet that amount of commits is necessary to continuously correct the shitload of mistakes coming from their "Hacking" culture.
They deal with this with automated code review tools like their Infer static analysis tool (http://fbinfer.com) as well as abstracting away the Objective-C with React Native.
Yeah that might work for programming mistakes, but not sure it can help with bad design choices.
Am I reading this right, there are interns and engineers running PHP in a debugger on production data? How else would he be able to memorialize his own account and trigger the email?
That is a good point, how did no-one else pick up on that???! It reminds me of engineers where I used to work, who would run through a £0.01 transaction on their credit card to check the changes to the charging system were still working...
I've seen talks with engineers that say this is this case. I think the one in particular was from @Scale. They've had the occasional issue with an intern dropping a table in production, but they said they haven't really run into issues with granting access to production data.
Would be interested to see some comparison with the Android app which seems to have most of the same functionality at 1/4 the size with 25MB [1] compared to the 100MB of iOS. Seems like either iOS team at FB has significant cruft laying around or iOS doesn't scale as well as Android and FB is hacking around that. Possibly a bit of both but think it would be an interesting comparison in addition to the war stories about hacking their entire tool chain.
[1] https://gigaom.com/2015/01/26/facebooks-new-android-app-goes...
[1] https://gigaom.com/2015/01/26/facebooks-new-android-app-goes...
iOS apps are shipped as fat binaries, so for one, part of that 100mb app is duplicated slices of binaries for armv7, arm64 and possibly, but not likely, armv7s.
Android apps enjoy bytecode that is getting JIT-ed on install.
Android apps enjoy bytecode that is getting JIT-ed on install.
Apple has introduced App Thinning in iOS 9 apps which should cut down on their size:
https://developer.apple.com/library/watchos/documentation/ID...
https://developer.apple.com/library/watchos/documentation/ID...
The feature was recently delayed, but it should help once it's ready
Is there any technical justification for this or it's purely economical/financial?
For what, fat binaries? That's pretty much what Mach-O format had from inception under NeXTSTEP, and survived until now. As it was said in the sibling comment, iOS 9 is introducing a way to cut exactly the slice you need for a particular device.
This was pretty interesting. It would have never occurred to me that XCode, Git, etc. couldn't handle as much as you throw at it, but it kind of makes sense how they explain it here. Still, I don't think I would like to actually work in that environment where the codebase changed so much so often and had so much dead and redundant code, so I guess I'm glad my interview with them didn't go terribly well (got asked questions by non tech-type person who was looking for keywords to obscure concepts as answers and didn't understand my conceptual but keyword-less responses).
I've worked with clients that had all their code in one repo at Github, and it took me 5 tries to even pull it down initially. You need not operate at Facebook scale to overwhelm git :)
> got asked questions by non tech-type person who was looking for keywords to obscure concepts as answers and didn't understand my conceptual but keyword-less responses
Simple fact of the matter is that you're going to run into those type of questions by non-technical people at any big software company with which you interview. It's on you to be able to smell out what they want and answer with something that demonstrates your abilities and gives them what they're looking for.
Simple fact of the matter is that you're going to run into those type of questions by non-technical people at any big software company with which you interview. It's on you to be able to smell out what they want and answer with something that demonstrates your abilities and gives them what they're looking for.
What keywords do you think the interviewer was looking for? Was it along the lines of "disruptive", "game-changing", "innovative", etc.?
I don't remember what all of them were, but I specifically remember one of them had the words "Toll Booth" in it, like "Toll Booth Blah", where blah is the key word I can't remember. It's some networking/communication concept, I think. The phone interview person had a fairly difficult accent too so I don't think I even heard the whole answer.
I've only ever encountered the term once online since. I just tried to Google for it again and I can't find the write combo of terms to find it ("toll booth networking? no.... toll booth computer science? no.....")
So I tried to answer that question with what happens on a concept level, and the person on the line said "No. The answer is 'Toll Booth Blah'. 'Toll Booh Blah'" and moved on to the next question. Like she was Alex Trebek and there was only one way to answer the question.
Meanwhile Google thought I was good enough to fly over for an in-person interview at around the same time, so I just wrote Facebook off as being terrible at hiring and moved on.
To date it's still the worst phone interview experience I've had, and I've had some awful ones.
I've only ever encountered the term once online since. I just tried to Google for it again and I can't find the write combo of terms to find it ("toll booth networking? no.... toll booth computer science? no.....")
So I tried to answer that question with what happens on a concept level, and the person on the line said "No. The answer is 'Toll Booth Blah'. 'Toll Booh Blah'" and moved on to the next question. Like she was Alex Trebek and there was only one way to answer the question.
Meanwhile Google thought I was good enough to fly over for an in-person interview at around the same time, so I just wrote Facebook off as being terrible at hiring and moved on.
To date it's still the worst phone interview experience I've had, and I've had some awful ones.
Is it Toll-Free Bridging?
Yep, that was it. Man, I really misremembered that. It's an Objective-C thing, which makes sense, because it was for an iOS position. I knew the idea behind it, but I had never encountered the term. I do remember I ran into it again while preparing for interviews again a year ago. I'm deep in .NET territory for work and Python for fun right now, so my iOS knowledge has atrophied a bit.
Yep, I lost an interview about ten years ago in part because I had not encountered the magic term "duck typing" yet. Never mind that I had be making file-like and string-io objects in Python for years by that time.
Don't get me started, tech interviewers are so bad...
Don't get me started, tech interviewers are so bad...
The unnecessary bashing of PHP, the lame justification for a sloppy architecture, the immense ego... I don't think hack culture should be used to justify asshole-ery.
So i guess maybe Apple didn't code their ios and ios apps using xcode, xcode-git, autolayout and coredata after all. Because their scale seems pretty big too.
Some sidenotes: I'm surprised to see so many custom libraries for Paper, an app which I had forgotten about. And another reminder that a lot of companies at scale are choosing mercurial over git.
(Another post about React Native)
Facebook engineers keep telling everyone that React Native is not used in the main Facebook app. I always wondered what is stopping them from start using it. Perhaps app size is a massive concern? Maybe all this tooling around their app is not really compatible with React Native?
Its interesting that they have a ton of legacy Obj-c code (considering that companies started to build apps not very long time ago, and they already re-wrote their app several times) I wonder if any other company is a situation like this well other than Apple.
Facebook engineers keep telling everyone that React Native is not used in the main Facebook app. I always wondered what is stopping them from start using it. Perhaps app size is a massive concern? Maybe all this tooling around their app is not really compatible with React Native?
Its interesting that they have a ton of legacy Obj-c code (considering that companies started to build apps not very long time ago, and they already re-wrote their app several times) I wonder if any other company is a situation like this well other than Apple.
It would probably be a huge pain to migrate all 18,000 classes, tests, tools, etc., over to a React Native app. Also JS has less fine grained tooling for performance profiling (try stepping through JS bytecode for performance analysis, it's a PITA). One day it might get there, but not today. If it works, why change it?
I am not talking about migrating the whole app but how about for new features? I noticed that they redesigned their friends request page, why not build that with react native?
The engineers who did that redesign didn't want to use react native or felt it wasn't a good choice. Just because one person at Facebook built a tool doesn't mean that everyone else at Facebook is suddenly required to use that tool where ever possible.
This was an awesome read. Thanks Simon!
This post is very interesting. News reported Facebook would create a simplified Android app for low end phone owners in Africa. I think They should do it for iOS too.
They already have, it's called Facebook Lite. You can't get it from the Play Store in the US, but I download an APK. It's nice.
I only really downloaded it so that I could get notifications, and now m.facebook.com can send them from Chrome, so I'm very happy to totally app-less.
I only really downloaded it so that I could get notifications, and now m.facebook.com can send them from Chrome, so I'm very happy to totally app-less.
I loved the anecdote starting on page 41 where he accidentally "memorializes" his Facebook profile thus triggering an email to his wife about his death.
Very interesting, even as a fan of native apps I can see the thought process behind doing it web based when you have to build so many tools for each platform.
Except it's mostly not web based.
Yes interestly enough there was a period when the front-end for mobile was html5 based but they since ditched that for greater (native) performance, in fact as this article reveals, a number of implementations!
As Synaesthesia notes, it was. I meant with processes and tools like the post describes building once sound like a good idea...
what is fb doing for iOS builds and compiles?
"Done is better than perfect"
"Move fast and break things"
I'm glad they're not holding any critical data like our personal information, what we do, what we think and where we are then!
If this set of slides embodies 'hacker culture' which it claims to, I am not a hacker, neither are my colleagues and I'm proud of that.
The whole thing stinks of arrogance hiding incompetence. I'm not putting anything near facebook, ever.
"Move fast and break things"
I'm glad they're not holding any critical data like our personal information, what we do, what we think and where we are then!
If this set of slides embodies 'hacker culture' which it claims to, I am not a hacker, neither are my colleagues and I'm proud of that.
The whole thing stinks of arrogance hiding incompetence. I'm not putting anything near facebook, ever.
To be fair, most of these specific signs have been removed now/changed (I was an intern there fall 2013). They actually talked about how that used to be a slogan, but breaking stuff isn't really acceptable anymore because facebook is so big. They still wanted us to move really fast though...
Facebook's new slogan is "Move fast with stable infrastructure."
From what I have heard they are now expecting their employees to still move fast, break stuff, but be ready to be penalized for it if you do. So, really, they are encouraging their employees to move at a steady pace with occasional hack nights when there is an upcoming release in the near-term. Facebook is growing old.
Trying to correct all the misunderstandings in this discussion would be a little soul crushing; but just picking this one comment as it seems well-intentioned and doesn't go anywhere near anything NDA-covered ;)
> break stuff, but be ready to be penalized
I've seen things break a few times, and the attitude is always "how do we fix this" and then "how do we fix the system to prevent this type of error in the future", never "whose fault is this"
> occasional hack
They seem to happen fairly often
> nights
They're mostly days, since a large number of people have families to go back to; some people stay late with beer and pizza and friends, because that's the kind of thing they enjoy doing, but it's an optional extra. (Technically the hack day is optional too -- if you're not interested, you can carry on as normal)
> when there is an upcoming release
Hacks aren't about video-game-industry style deadline crunches - quite the opposite; a hack day is when you pause your regular work (barring emergencies, ofc) in order to explore and experiment with things you're interested in but wouldn't normally work on.
> in the near-term
Releases are always near (like, multiple times per day).
> break stuff, but be ready to be penalized
I've seen things break a few times, and the attitude is always "how do we fix this" and then "how do we fix the system to prevent this type of error in the future", never "whose fault is this"
> occasional hack
They seem to happen fairly often
> nights
They're mostly days, since a large number of people have families to go back to; some people stay late with beer and pizza and friends, because that's the kind of thing they enjoy doing, but it's an optional extra. (Technically the hack day is optional too -- if you're not interested, you can carry on as normal)
> when there is an upcoming release
Hacks aren't about video-game-industry style deadline crunches - quite the opposite; a hack day is when you pause your regular work (barring emergencies, ofc) in order to explore and experiment with things you're interested in but wouldn't normally work on.
> in the near-term
Releases are always near (like, multiple times per day).
Does Facebook pay overtime for "hack nights"?
No, but they make up for it by consistently throwing piles of money at peoples' feet which dwarf the compensation of all but a handful of other jobs.
High profile companies seem to have a privilege complex I.e. you're lucky to be working for them.
It seems like with such a strong engineering team, Facebook should have really built their own phone. It's impressive that they re-wrote a lot of the major components the iOS development process and optimized them for their scale. I think a Facebook style development platform on it's own hardware would be an interesting contrast to Google's and Apple's solutions. One challenge with learning this type of development stack is that you're almost not even learning traditional iOS development so your skills aren't truly transferrable.
> It seems like with such a strong engineering team, Facebook should have really built their own phone.
They actually tried not too long ago... [0]
[0] http://www.cnet.com/news/heres-why-the-facebook-phone-floppe...
They actually tried not too long ago... [0]
[0] http://www.cnet.com/news/heres-why-the-facebook-phone-floppe...
I'd say Facebook's emphasis on developer tools shows they'd be more committed to go down the route of building custom hardware for their devs like mouses, keyboards, laptops, workstations, etc. As opposed to building hardware for end-users.
So, FB app is bloated because they choose to be sloppy is what I read. Wonder how sustainable this is over the long term?
It's not sustainable.
But since software developers have been dealt a good hand (demand-wise) in these decades, we get away with it. And call it "hacker culture", so it sounds hip and even correct.
But since software developers have been dealt a good hand (demand-wise) in these decades, we get away with it. And call it "hacker culture", so it sounds hip and even correct.
Actually, I'd counter that it is sustainable -- for Facebook. They have so much user data and so much ad revenue that they can afford to throw engineers and money at the problem. Big companies can work hard and make things work, smaller shops have to work efficiently out of necessity.
Yes indeed. From a slightly different perspective, assuming every good dev tries to be as efficient as they can be, even assuming they are, but with varying degrees of success due to various factors, a big company can hire lots of good people and only needs to get lucky and hit the mark occasionally. A small company on the other hand has to get lucky pretty much every single time.
This strategy was hugely successful for Microsoft since the early days of Windows NT 3.1: “What Intel giveth, Microsoft taketh away.”
The performance advancements on the hardware side of the platform are almost entirely consumed by the ever-increasing complexity of the most popular app on the software side.
The performance advancements on the hardware side of the platform are almost entirely consumed by the ever-increasing complexity of the most popular app on the software side.
That may be successful for the software companies. But pity the user who's stuck with old hardware and wants to know why used computers don't last as long as used cars. Maybe when Moore's Law finally becomes invalid...
Actually, the hacks they used make the app work better on older hardware.
As another example, the proprietary office formats contained a lot of hacks specifically designed to make the files load faster on older PCs. Newer PCs were able to compensate with the complexity because they were faster.
Yep. 2005 G5 powermac. Totally usable machine. Can get a dual 2.5GHz boz with 4Gb of RAM for equivalent $40 in the UK.
But no, it's stuck on OSX 10.5...
But no, it's stuck on OSX 10.5...
Sounds like a job for Debian!
Debian was a complete dick on my old ppc g4. It sort of put me off. The was going to try FreeBSD but I have to persuade the wife to let me buy a g5 yet as they're large, noisy and entirely unnecessary. I just want one to play with and go ooh and aah and stuff :)
I did the same with old SPARC kit a number of years ago too.
I did the same with old SPARC kit a number of years ago too.
Product-wise, it's likely that in the long term they're going to break up the FB app into a federation of related apps. They already did that with Messenger.
Sloppy?! Those guys have needs the majority of us will never need to deal with. Did you read the presentation?
If anything they're the opposite of sloppy.
If anything they're the opposite of sloppy.
The parent is not referring to those needs (which are legitimate). The presentation talks a lot about the hacking culture; how there is a lot of duplicate code, a lot of dead code, a lot of "temporary" code.
We call these things technical debt. But it sounds like they just never fix them. Refactors never happen.
We call these things technical debt. But it sounds like they just never fix them. Refactors never happen.
Eventually you just hear "New version of our app rewritten from the ground up" when it all becomes one big pile of technical debt.
I sincerely doubt the fact that there are companies which thoroughly fix "technical debt".
They don't call it debt. It is simply their way of keeping their agility with a huge number of developers. If the flip side is junk that mostly impacts install space, I don't see it as a bad trade-off.
What alternative way would you suggest to keep your mega team productive and shipping features?
What alternative way would you suggest to keep your mega team productive and shipping features?
You can't avoid fixing tech debt. Eventually it impedes your ability to be productive. It happens, to some degree, to every software project. Someone else joked about "we wrote the app from the ground up" but that really does happen. You're not "shipping features" if you're rewriting the app.
Hiring a team whose sole job description is garbage cleanup?
Productivity != Total Output.
1. Having a core team who's job it is to maintain the app and clean up the debt.
2. I seriously have to question that they absolutely need that pace. I don't buy it for a second that any of this is necessary or better than the alternative.
2. I seriously have to question that they absolutely need that pace. I don't buy it for a second that any of this is necessary or better than the alternative.
Their needs don't explain all the redundant and unused code. It's slop, plain and simple.
It's not. It's just decoupling. Redundant code exists because different groups of people needed the same thing in different contexts (with probably slightly different requirements), and co-ordinating to keep this stuff centralised is hard, and most of all time-consuming.
This has two clear advantages. You don't need to fill a form every time you change the smallest of functions and when you mess up, you don't impact everyone else. Honestly, I don't get the negativity.
EDIT: I don't mention dead code because that can be, if really needed, removed by a tool so I think the point is moot.
This has two clear advantages. You don't need to fill a form every time you change the smallest of functions and when you mess up, you don't impact everyone else. Honestly, I don't get the negativity.
EDIT: I don't mention dead code because that can be, if really needed, removed by a tool so I think the point is moot.
> I don't mention dead code because that can be, if really needed, removed by a tool so I think the point is moot.
It can easily be cleaned up in an automated fashion and they don't do it. That's exactly what everyone means by slop. If you could pick up the clothes on your floor but don't, it's not because you're a hacker.
Update: not only do they not clean it up, they ship this shit to a billion people. Talk about lazy.
It can easily be cleaned up in an automated fashion and they don't do it. That's exactly what everyone means by slop. If you could pick up the clothes on your floor but don't, it's not because you're a hacker.
Update: not only do they not clean it up, they ship this shit to a billion people. Talk about lazy.
Their needs are because they're sloppy. Having an overarching Core team who's job is to maintain the app and guide the architecture would cut down on what they need immensely.
This is truly a superb read. Don't just read the comments!
There is absolutely no reason for the Facebook app to be so large and have so many classes. The app itself doesn't do that much.