The Largest Rails Application Loses Ground To PHP Competitor Due to Frequent Outages(techcrunch.com)
techcrunch.com
The Largest Rails Application Loses Ground To PHP Competitor Due to Frequent Outages
http://www.techcrunch.com/2007/05/17/making-the-switch-from-twitter-to-jaiku/
5 comments
I think this illustrates the huge importance of uptime, even for free services. On the internet there is (almost) always a choice and people will pick even a lesser competitor over you, if you can't keep your damn servers alive.
I'm not trying to knock on anyone's development platform of choice, but how can anyone expect unproven software to perform at the kind of scale that some of these web apps are being asked to perform at? Lisp is a rather old and finely tuned language, but in Ansi Common Lisp, Paul Graham even suggests re-writing frequently-run sections of code in C or assembler. If a highly-optimized language needs to be profiled and tuned, how can that be avoided with a language like Python or Ruby?
Of course, there are issues such as SQL queries and system/hardware configurations that come into effect as well. The point is this: at some point, when you are trying to grow a product to such a massive scale, you need to expend some energy on optimization. If you just continue growing things in the same way you built your initial prototype, you're doomed to failure.
Of course, there are issues such as SQL queries and system/hardware configurations that come into effect as well. The point is this: at some point, when you are trying to grow a product to such a massive scale, you need to expend some energy on optimization. If you just continue growing things in the same way you built your initial prototype, you're doomed to failure.
There's a difference between optimization and reliability.
For instance, Erlang isn't the fastest thing out there, but you can make some extremely reliable systems with it.
I still think Rails is a winner because of how fast it lets you develop the application. Most of us will probably never see the massive popularity spike that something like twitter has. So, better to optimize for development time, and put code speed optimization off until it's necessary.
For instance, Erlang isn't the fastest thing out there, but you can make some extremely reliable systems with it.
I still think Rails is a winner because of how fast it lets you develop the application. Most of us will probably never see the massive popularity spike that something like twitter has. So, better to optimize for development time, and put code speed optimization off until it's necessary.
True...Optimizing for getting to that popularity level fast is the wise thing to do, and so the most productive tool to develop in is the best one.
But...there are major costs to a lack of reliability, and they can rear their head before you reach that level of popularity. If you're not able to stay up during that all important first crunch or slashdotting or digg...you won't ever make it to the sustained popularity and growth that requires scaling up. RoR does seem to have some problems with that.
Ruby is the slowest of the most popular scripting languages, by most measures. RoR is pathological with databases in some circumstances (not common cases, admittedly). And, from what I can tell most RoR stacks (Apache+fcgi, Mongrel, ngenix, Lighty, etc.) have one or more pretty serious flaws when scaling up.
I'd still consider using it, since it's so much damned fun to write Ruby code.
But...there are major costs to a lack of reliability, and they can rear their head before you reach that level of popularity. If you're not able to stay up during that all important first crunch or slashdotting or digg...you won't ever make it to the sustained popularity and growth that requires scaling up. RoR does seem to have some problems with that.
Ruby is the slowest of the most popular scripting languages, by most measures. RoR is pathological with databases in some circumstances (not common cases, admittedly). And, from what I can tell most RoR stacks (Apache+fcgi, Mongrel, ngenix, Lighty, etc.) have one or more pretty serious flaws when scaling up.
I'd still consider using it, since it's so much damned fun to write Ruby code.
Ahh, you're right. I should have said, "you need to expend some energy on optimization and ensuring reliability." I guess I was making the assumption that the software was basically working, but failing under heavy load, which seems to be an optimization problem. I let it be given that the software is correct, but perhaps that is not a safe thing to do.
And I totally agree that Rails et al are great development platforms to get up and running quickly, but when that software begins to fail with increasing traffic, it's time to first reconsider your own code (algorithms and bulletproofiness (as Colbert might say)), and if that doesn't help, to consider rewriting at least parts in something else.
I can't imagine the call stack for a single Rails function call (I'm actually not all that familiar with Rails). At the very least, it might be good to migrate some code from Rails to straight Ruby, and some of the auto-generated stuff to hand-tuned SQL. That's all I was trying to say.
And I totally agree that Rails et al are great development platforms to get up and running quickly, but when that software begins to fail with increasing traffic, it's time to first reconsider your own code (algorithms and bulletproofiness (as Colbert might say)), and if that doesn't help, to consider rewriting at least parts in something else.
I can't imagine the call stack for a single Rails function call (I'm actually not all that familiar with Rails). At the very least, it might be good to migrate some code from Rails to straight Ruby, and some of the auto-generated stuff to hand-tuned SQL. That's all I was trying to say.
I think you're making the mistake of thinking just because they couldn't make Rails scale that no one could have. They probably have guys working on the problem that are early employees -- not necessarily the best of the best. That said, I don't really know too many details about who they have working on it.
That was the purpose of my second paragraph, but maybe I didn't get the point across enough. I don't think Rails is bad, not at all. I think Rails is great for getting projects up and running quickly, and for easily adding to the projects later on. You simply need to expend some energy on optimization (and reliability) once you find yourself growing to sizes that test your current code base. This could involve writing some frequently used code in a low-level language, rewriting a Rails function you're using which does more than you really need into straight Ruby, or reconsidering some of your SQL queries and handwriting some of those that are automatically handled.
All I'm trying to say is that no matter who you are, at some point in the scaling process, you are going to have to revisit your code and make improvements. The great thing about Ruby, Python, Lisp, etc is the ability to throw a bunch of code down and get something running quickly. This code is inherently not optimal, and this is not a bad thing. Once you start to see problems, however, it's time to profile and re-think some of the things you are doing in order to improve your software overall.
All I'm trying to say is that no matter who you are, at some point in the scaling process, you are going to have to revisit your code and make improvements. The great thing about Ruby, Python, Lisp, etc is the ability to throw a bunch of code down and get something running quickly. This code is inherently not optimal, and this is not a bad thing. Once you start to see problems, however, it's time to profile and re-think some of the things you are doing in order to improve your software overall.
Adding to the point you made about rewriting code - it's really easy to write inline C with Ruby. DHH has spoken about how 37signals found some performance "hotspots" with Basecamp (?) and simply dropped down into C and did a line-by-line translation to fix their issues.
This is exactly the kind of thing I was talking about. Thanks for the good concrete example (most of what I said was hand-waving :-).
One of the nice features in Common Lisp is the ability to optionally declare types on objects to make them static. This is very convenient when you want to optimize something on-the-fly; just find the bottleneck and add a type declarator to it. This can go a long way before you drop into C.
I completely agree. This is a feature I would love to see in more languages, as it's such a no-brainer optimization that just about any language could benefit from its use.
I don't see any reason to assume that Jaiku will scale better than Twitter a priori.
I would hope it's running more reliably than Twitter know since it has only a fraction of the users.
I would hope it's running more reliably than Twitter know since it has only a fraction of the users.
I think this is a really important point. Here:
http://www.alexa.com/data/details/traffic_details?site0=twitter.com&site1=jaiku.com&site2=&site3=&site4=&y=r&z=1&h=300&w=610&range=6m&size=Medium&url=twitter.com
http://snapshot.compete.com/twitter.com+jaiku.com
If you believe the public stats, Jaiku is 6x-15x smaller that Twitter.
If you believe the public stats, Jaiku is 6x-15x smaller that Twitter.
Maybe they should call the Scribd guys, they're doing more traffic than I thought: http://snapshot.compete.com/scribd.com+twitter.com
The question to ask is whether a competitor can do something better. I don't use twitter, but it sounds like any competitor would also have to be doing a similar number of database queries.
So, any competitor would still have to go through what twitter has, so twitter is still ahead by now focusing on the scalability.
Twitter has to move people to a non-rails platform as soon as possible. The whole point of rails or visual basic 6 is to develop a prototype really quickly and change your ideas to find the one that works. If the problem now is the speed, just pretend you're a competitor and do what the competitor would do in your situation. A competitor wouldn't be recreating the failed versions of Twitter; they would just copy the current idea in C or whatever the fastest method is, since the business idea and execution is more valuable now than flexibility, now that the business idea has been discovered and twitter doesn't have to spend time making so much changes. Except, Visual Basic 6 programs are very fast on modern computers, and they run on each user's computer separately, so you wouldn't want to rewrite them for Windows if it already exists and works.