Ask YC: How hard is it to keep a service like twitter online
5 comments
Not all high-traffic websites are equal. Youtube has way more visitors than Twitter, but is a far simpler site to scale.
Twitter is a particularly difficult problem, because unlike most high-traffic sites the number of writes is pretty close to the number of reads. The latency expected by users is also very low - people are carrying on conversations over it after all, and these two factors combined mean that there's very little that can be done in the way of caching. Add this the complex relationships between all the different users and you've got a difficult site to scale to the sort of volumes they're experiencing.
If the guys at Twitter want to offer me a job, I've got a few ideas though ;)
Twitter is a particularly difficult problem, because unlike most high-traffic sites the number of writes is pretty close to the number of reads. The latency expected by users is also very low - people are carrying on conversations over it after all, and these two factors combined mean that there's very little that can be done in the way of caching. Add this the complex relationships between all the different users and you've got a difficult site to scale to the sort of volumes they're experiencing.
If the guys at Twitter want to offer me a job, I've got a few ideas though ;)
Twitter is a particularly difficult problem
Excuse me? Twitter, at its core, is a large-scale pub/sub system with a bunch of frontends. Such systems are well understood and, frankly, quite trivial as far as the non-existant constraints of the twitter-app are concerned. After all no ordering guarantees or fancy routing are needed and obviously they don't give a damn about fault tolerance or latency either.
Their constant failure to keep that thing online just screams incompetency very loudly...
There are plenty of mature building blocks available in the OSS world to implement such a system. RabbitMQ and the spread toolkit come to mind.
Or, if you have the dough, you could buy a shrinkwrapped solution from TIBCO, Solace or the like. The latter will even come with fancy functionality and availability guarantees in writing because those are normally deployed in "mission critical" industrial systems, such as logistics, banks, stock exchanges.
Now, twitter is not SWIFT and doesn't need five nines of uptime. But saying that this is a "particularly difficult problem" is, sorry, ridiculous.
Excuse me? Twitter, at its core, is a large-scale pub/sub system with a bunch of frontends. Such systems are well understood and, frankly, quite trivial as far as the non-existant constraints of the twitter-app are concerned. After all no ordering guarantees or fancy routing are needed and obviously they don't give a damn about fault tolerance or latency either.
Their constant failure to keep that thing online just screams incompetency very loudly...
There are plenty of mature building blocks available in the OSS world to implement such a system. RabbitMQ and the spread toolkit come to mind.
Or, if you have the dough, you could buy a shrinkwrapped solution from TIBCO, Solace or the like. The latter will even come with fancy functionality and availability guarantees in writing because those are normally deployed in "mission critical" industrial systems, such as logistics, banks, stock exchanges.
Now, twitter is not SWIFT and doesn't need five nines of uptime. But saying that this is a "particularly difficult problem" is, sorry, ridiculous.
There is an old saying, those who were asleep in CS 101 are doomed to be mocked on the Internet forever :-)
IMO, it's not a hard problem, they a just incompetent. Edit: For more constructive criticism:
1) Don't require a DATABASE as part of a real time system.
2) If it's not in RAM your doing it wrong.
the number of writes is pretty close to the number of reads
This is only a "problem" if your database is MySQL. People have been doing this kind of workload on the major commercial database for decades.
This is only a "problem" if your database is MySQL. People have been doing this kind of workload on the major commercial database for decades.
Here is more about Twitter's architecture, I guess it has changed a lot right now. Also check out the presentation by Blaine Cook, Architect of twitter.
http://highscalability.com/scaling-twitter-making-twitter-10...
http://www.slideshare.net/Blaine/scaling-twitter
http://highscalability.com/scaling-twitter-making-twitter-10...
http://www.slideshare.net/Blaine/scaling-twitter
They essentially deal with millions of real-time messages, each of which have a fairly complicated relationship tree. These issues aren't so much a database problem as a message passing and queuing problem.
Why can't you just pass this to X number of servers?
The simple answer is: In the beginning they thought Twitter would more like Livejournal and architected accordingly (not that LJ is the pinnacle of good architecture or anything but even so). Instead it turned out to be more like IM and rather than starting again they decided to hack it into working.
Although the downtime must be frustrating for users and the temptation to build the shiny-new-system-that-will-solve-everything nearly irresistible for developers, I think they probably made the right decision not to do this. They have a runaway hit on their hands. That makes the risks associated with rewrite way higher (and they're already way high). One thing that people often do is start a "next-gen" project with an "architecture" team to figure out the new thing while a maintenance team (often lower-status within the organization) works on the existing product. This is so consistently bad an idea, it's pretty much a red flag all by itself. If I were ever going to rewrite a successful system I'd insist that it be the original team doing it.
I also wonder whether Twitter's downtime doesn't psychologically reinforce the sense that everybody's using it so it must be valuable. Scarcity and all that.
One thing about Twitter does make me pause. Haven't they hired tons of programmers? What are they all doing?
I also wonder whether Twitter's downtime doesn't psychologically reinforce the sense that everybody's using it so it must be valuable. Scarcity and all that.
One thing about Twitter does make me pause. Haven't they hired tons of programmers? What are they all doing?
Interesting, so its something to actually learn from in regards to initial design flexibility.
But if I might ask? Would re-designing the entire architecture to accommodate for its current use be a wise idea to solve their downtime issue? I am just curious
But if I might ask? Would re-designing the entire architecture to accommodate for its current use be a wise idea to solve their downtime issue? I am just curious
Hm.. isn't friendfeed similar to twitter? They don't seem to get (any) downtime. Also, there's some Indian twitter-like service which apparently has higher traffic volumes and no downtime either.
If all this is correct, then it's probably not "that hard" to keep something like twitter with an "acceptable" uptime.
If all this is correct, then it's probably not "that hard" to keep something like twitter with an "acceptable" uptime.
I once launched a Facebook IM application using AjaxIM. Everything was beautiful for about 10 days. I went from 90 users to 18K users in 21 days and the whole application came to a HALT. IMs were taking 5 seconds to get to the other end. I, for sure, did not have the technical ability to make it work and I decided to not invest money into it. We were dumping the DB of IMs every 3 hours, but with IM you are dealing with hundreds of thousands of real-time messages. This is extremely hard. Add to this the fact that each Twitter account has a different graph and your messages go to several people.
Not having worked with high-traffic websites I've been wondering why they're having such problems. I would imagine that if you have money (which they do) you could just throw more servers and a good SQL consultant after the problem. Is there a problem specific to their architecture that is hard to scale? If so what is the constraint?
I'm just puzzled that they haven't solved it, and hope they do.