Web Framework Benchmarks(techempower.com)
techempower.com
Web Framework Benchmarks
http://www.techempower.com/blog/2013/03/28/framework-benchmarks/
396 comments
"Among the many factors to consider when choosing a web development framework, raw performance is easy to objectively measure."
Oh really? Then why did Zed write such an angry rant about how you are doing it wrong?
http://zedshaw.com/essays/programmer_stats.html
Can we please see some standard deviations, at least?
Oh really? Then why did Zed write such an angry rant about how you are doing it wrong?
http://zedshaw.com/essays/programmer_stats.html
Can we please see some standard deviations, at least?
I'm torn about this. On the one hand, while I've known my framework of choice (Rails) is slow, I didn't know how much slower it could be in the grand scheme of things. But on the other hand, I'm more shocked by the difference between EC2 and dedicated hardware (10x improvement with rails), and even 89 requests per second (20 query benchmark on EC2) is still a decent amount of traffic. (Plus this doesn't count any optimizations I would make anyway, like caching).
Either way good architectures usually optimize the high traffic or high CPU areas anyway away from a scripted language.
Thanks for the really informative post! Go seems to be a good balance as a high performance language without having to go back to my traumatic Java days.
Either way good architectures usually optimize the high traffic or high CPU areas anyway away from a scripted language.
Thanks for the really informative post! Go seems to be a good balance as a high performance language without having to go back to my traumatic Java days.
I worked with Pat (pfalls) on this effort. He pulled the benchmarks together and built the script to automate the tests. We aimed to deploy each framework/platform according to best-practices for a production environment and then stress test common operations: JSON serialization of objects and database connectivity. We were surprised by the wide spectrum of performance we observed and hope that this interesting to you as well. Four orders of magnitude in one of our tests!
If you have any questions or see something we stupid we did, please let us know. We'd like to correct any mistakes straight away, especially since we're certainly not experts on all of these frameworks and platforms.
If you have any questions or see something we stupid we did, please let us know. We'd like to correct any mistakes straight away, especially since we're certainly not experts on all of these frameworks and platforms.
Where is ASP.Net MVC? Odd that you list obscure frameworks like Wicket and leave out one of The Big Four frameworks.
(The big four in my book are: ASP.Net MVC, Rails, Django and CakePHP)
(The big four in my book are: ASP.Net MVC, Rails, Django and CakePHP)
I'd like to see how .Net MVC would compare. I realize you'd have to spin it up on a Windows EC2 instance and there would definitely be some variance in the performance of that box vs. the nix EC2 instances but I'd still be interested in seeing how it fares in comparison.
> This exercise aims to provide a "baseline" for performance across the variety of frameworks. By baseline we mean the starting point, from which any real-world application's performance can only get worse.
I disagree with the implication here (that this is a good point for comparison because "real-world application's performance can only get worse."). Yes it can only get worse but how much worse (per unit of "features") is both significant and unaddressed.
This isn't the best example but look at the gap between the top and bottom of the scale in the Database access test (single query) and Database access test (multiple queries) charts: In the first, Gemini is ~340x faster than Cake, in the second, only ~23x faster. There is still a big gap but it closed by an order of magnitude once you stepped past the most trivial possible DB access test.
So nodejs or php-raw is faster than cake at a single DB access, but what about when you create a real world scenario with authentication, requirement to be able to update features faster (i.e. use an ORM), env. portability requirement, etc.? It seems to me this would look like a little slower, a little slower, a little slower in the {raw} versions, and already included, already included, already included in Rails or Cake. The full featured frameworks take a lot of their performance penalty up-front, with less of a hit as features are added (maybe? :P).
My point is that it's not reasonable to assume that hackernews-benchmarks will actually reflect production use. That said I think the article is cool, and agree that it's good to keep framework authors' feet to the fire regarding performance!
I disagree with the implication here (that this is a good point for comparison because "real-world application's performance can only get worse."). Yes it can only get worse but how much worse (per unit of "features") is both significant and unaddressed.
This isn't the best example but look at the gap between the top and bottom of the scale in the Database access test (single query) and Database access test (multiple queries) charts: In the first, Gemini is ~340x faster than Cake, in the second, only ~23x faster. There is still a big gap but it closed by an order of magnitude once you stepped past the most trivial possible DB access test.
So nodejs or php-raw is faster than cake at a single DB access, but what about when you create a real world scenario with authentication, requirement to be able to update features faster (i.e. use an ORM), env. portability requirement, etc.? It seems to me this would look like a little slower, a little slower, a little slower in the {raw} versions, and already included, already included, already included in Rails or Cake. The full featured frameworks take a lot of their performance penalty up-front, with less of a hit as features are added (maybe? :P).
My point is that it's not reasonable to assume that hackernews-benchmarks will actually reflect production use. That said I think the article is cool, and agree that it's good to keep framework authors' feet to the fire regarding performance!
Would've loved to see http://servicestack.net on this list which has great performance on .NET and Mono: https://github.com/ServiceStack/ServiceStack/wiki/Real-world...
And also maintains .NET's fastest JSON and Text Serializers: http://theburningmonk.com/2011/11/performance-test-json-seri...
And also maintains .NET's fastest JSON and Text Serializers: http://theburningmonk.com/2011/11/performance-test-json-seri...
pfalls - amazingly, I spent the last 2 days of my holiday doing the same thing for a future open source project. I was just stumped when I saw you guys did the same (could have saved me a couple of days!)
I wanted to find the leanest Web framework on any kind of platform; but the difference from your approach - I already knew the kind of code that would run on it.
I tested: Go, Java (servlet, dropwizard), Scala (scalatra), Ruby, Node.js (connect).
For me it was:
* Scala
* Java
* Clojure (equal to Java - big surprise here)
* Node.js
* Go (almost equal to Node.js)
* Ruby (far far down)
Scala took the lead with amazing results. More over, a good metric was latency which Scala was the only one to take micro-second resolution.
I'm not a fan of Scala because of its surrounding tools, which is why I'm still considering going for either Clojure or Node.js.
I think the most surprising positively was Clojure, being that it is a dynamic language. And most surprising negatively was Go - by itself is impressive, but when given real work (Web handling, Redis/mongodb) goes bad quickly. Happy to see this correlates with your findings too; I'm assuming this is a symptom of library maturity..?
I'd be happy to see how Scala fares on your tests.
You've done an awesome job!
I wanted to find the leanest Web framework on any kind of platform; but the difference from your approach - I already knew the kind of code that would run on it.
I tested: Go, Java (servlet, dropwizard), Scala (scalatra), Ruby, Node.js (connect).
For me it was:
* Scala
* Java
* Clojure (equal to Java - big surprise here)
* Node.js
* Go (almost equal to Node.js)
* Ruby (far far down)
Scala took the lead with amazing results. More over, a good metric was latency which Scala was the only one to take micro-second resolution.
I'm not a fan of Scala because of its surrounding tools, which is why I'm still considering going for either Clojure or Node.js.
I think the most surprising positively was Clojure, being that it is a dynamic language. And most surprising negatively was Go - by itself is impressive, but when given real work (Web handling, Redis/mongodb) goes bad quickly. Happy to see this correlates with your findings too; I'm assuming this is a symptom of library maturity..?
I'd be happy to see how Scala fares on your tests.
You've done an awesome job!
1) The Python version has some basic newbie coding errors. This sort of code is what Python programmers call "Java written in Python". It may be a valid algorithm in Java, but it's the wrong way to do it in Python. Code like this will work, but it will be slow. Depending on the size of "queries", you are potentially allocating gobs of memory in two different places for no reason, and then throwing it away without using it. I wouldn't be surprised if the examples in other languages had similar problems.
2) The JSON serializer in Django 1.4 uses a method which is known to be very slow, but which is easily portable across different platforms and works with older versions of Python. They no doubt included for easy bundling. In a real application you would probably want to simply use the normal JSON serializer from the standard library (which is many times faster).
3) The examples are little more than "hello world". I did some benchmark tests with several Python async frameworks, Pypy, and Node.js for an application I was working on. With small JSON objects there wasn't much difference in performance. Once you started using large JSON objects the performance lines for all versions were indistinguishable from each other. The performance bottlenecks were in libraries, and those standard libraries were all written in 'C', so interpreter versus compiler versus JIT made little difference.
4) The problem with "toy" examples is that in real life there are two performance factors which must be taken into account. Think of as y = mx + b. With a toy example you are probably only measuring "b". With most real life applications it's "m" that matters. There are often different optimization approaches that are best for varying ratios of "b" and "m". You have to know your application intimately and benchmark using data which is realistic for that application.
Python has a reputation for being "easy to learn". However, it is "easy" in the sense of being able to hack something together that works without knowing very much. There can be several different ways of doing things and doing it one way versus another way can mean a difference in performance of several orders of magnitude. The same may be true for some of the other languages, but I haven't examined them in enough detail to say.
2) The JSON serializer in Django 1.4 uses a method which is known to be very slow, but which is easily portable across different platforms and works with older versions of Python. They no doubt included for easy bundling. In a real application you would probably want to simply use the normal JSON serializer from the standard library (which is many times faster).
3) The examples are little more than "hello world". I did some benchmark tests with several Python async frameworks, Pypy, and Node.js for an application I was working on. With small JSON objects there wasn't much difference in performance. Once you started using large JSON objects the performance lines for all versions were indistinguishable from each other. The performance bottlenecks were in libraries, and those standard libraries were all written in 'C', so interpreter versus compiler versus JIT made little difference.
4) The problem with "toy" examples is that in real life there are two performance factors which must be taken into account. Think of as y = mx + b. With a toy example you are probably only measuring "b". With most real life applications it's "m" that matters. There are often different optimization approaches that are best for varying ratios of "b" and "m". You have to know your application intimately and benchmark using data which is realistic for that application.
Python has a reputation for being "easy to learn". However, it is "easy" in the sense of being able to hack something together that works without knowing very much. There can be several different ways of doing things and doing it one way versus another way can mean a difference in performance of several orders of magnitude. The same may be true for some of the other languages, but I haven't examined them in enough detail to say.
Numerous irregularities plus a strong vested interest in the JVM make me doubt they have given adequate shrift to Go, here.
Given the amount of interest in Haskell and Yesod around here, it is strange that it is missing.
Given the amount of interest in Haskell and Yesod around here, it is strange that it is missing.
Would love to see how these results compare to some of the web frameworks for concurrent functional languages like Erlang/Haskell: Nitrogen, Chicago Boss, Snap, Yesod, etc.
This is exactly why I decided to use PHP for my startup. I have something along these lines that I hope to blog about in the coming weeks (I tested php-fpm on nginx/go/node.js/silk.js and php won by a landslide when it came to speed).
I would love to see php-fpm on nginx included in this test.
I would love to see php-fpm on nginx included in this test.
As a Rails developer and admirer, this is eye-opening. I love the framework (and Ruby especially), but these numbers bear some serious consideration.
30-50x performance difference gets really... real, no? The standard refrain of "throw more hardware at it" must reconcile with the fact that a factor of 30-50x means real dollars for the same amount of load. Is the developer productivity really that much greater?
30-50x performance difference gets really... real, no? The standard refrain of "throw more hardware at it" must reconcile with the fact that a factor of 30-50x means real dollars for the same amount of load. Is the developer productivity really that much greater?
How about Lift? Btw, the play framework you tested is Java or Scala based?
Either way, I'm shocked to see Play perform so slow comparatively. Although it's easily 10x faster than rails on most tests, I'm shocked to see Node.js faster than Play! (by 2x in most cases) Wow!!
Maybe Node.js critics should start appreciating it after all..
Either way, I'm shocked to see Play perform so slow comparatively. Although it's easily 10x faster than rails on most tests, I'm shocked to see Node.js faster than Play! (by 2x in most cases) Wow!!
Maybe Node.js critics should start appreciating it after all..
This seems to be a nice benchmark. For the Python group, I would suggest two things: (1) include a lightweight framework like Bottle, and (2) Try pypy.
I'm most surprised that PHP seems to be around an order of magnitude faster than Ruby on Rails, I knew it was faster, but didn't think it would be that much.
What kind of server did you guys use for your rails test? Thin, Puma, Unicorn? Are you sure you ran it in production environment?
Update:
Looks like passenger in development mode. Good job you benchmarked a web server that no one uses wile reloading all code between requests.
Update2:
Ok it seems to run in production mode but still, passenger is not an idiomatic choice.
Update:
Looks like passenger in development mode. Good job you benchmarked a web server that no one uses wile reloading all code between requests.
Update2:
Ok it seems to run in production mode but still, passenger is not an idiomatic choice.
Interesting - C# should have also been added. Java still rules the roost. If you should absolutely have a scripting language - old php with warts is better. More magic in the framework - more abstraction and indirection and code inefficiency - that can be the spoil port when it comes to performance.
What I can take from this is that when you use ORM, it slows things down considerably. Also, looking at rails example, you didn't use active record, which is really wrong.
I think you should tweak your tests to use more real world like examples. I realize it would be hard to do this across frameworks.
Like let's have a database query pull user record from 100,000 users by username. And maybe do md5 on password.
I think you should tweak your tests to use more real world like examples. I realize it would be hard to do this across frameworks.
Like let's have a database query pull user record from 100,000 users by username. And maybe do md5 on password.
How can you write a web framework benchmark and not include some of the non mainstream languages with probably the most performant frameworks like Erlang (Cowboy, Mochiweb), Haskell (Yesod, Snap Framework)? That's just wrong; anyway.
There is a huge difference between raw PHP and CakePHP. I'd be curious to see other PHP frameworks (such as Zend, or Slim) in there-- is Cake just particularly slow, or is that simply what happens when you have a PHP framework?
Utter crap.
"Sadly Django provides no connection pooling and in fact closes and re-opens a connection for every request. All the other tests use pooling."
But it's free, open-source software and we provide asynchronous database connection pooling for Postgres SQL:
https://github.com/iiilx/django-psycopg2-pool
"Sadly Django provides no connection pooling and in fact closes and re-opens a connection for every request. All the other tests use pooling."
But it's free, open-source software and we provide asynchronous database connection pooling for Postgres SQL:
https://github.com/iiilx/django-psycopg2-pool
I'm curious to see Django results when using the gevent worker for gunicorn. For these type of quick JSON calls, you can see huge performance increases.
I Will love to see dotNet (C#) incluid in this test.
Asp.Net WebAPI (Synchronous and Asycnhrounous)
Asp.Net MVC (Synchronous and Asycnhrounous)
Asp.Net HTTP Handlers (Synchronous and Asycnhrounous)
haha, all those hipster developers using rails can now eat the php guys shorts :)
Seriously though, this isn't news to anyone that does this professionally. The further up the abstraction curve you climb, the less performant the code will be. Ease of development vs run-time performance.
Seriously though, this isn't news to anyone that does this professionally. The further up the abstraction curve you climb, the less performant the code will be. Ease of development vs run-time performance.
What I'd love to see paired with this data is a cost comparison. At what scale does performance of ruby/python/php become cost prohibitive? Twitter made the move from Ruby to Java some years back, did they ever post a comparison of their numbers before and after?
Also, the difference between EC2 and local i7 hardware is glaringly obvious. At what scale does owning the server hardware become imperative?
I know these questions are beyond the scope of a performance review, but inquiring minds would like to know.
Also, the difference between EC2 and local i7 hardware is glaringly obvious. At what scale does owning the server hardware become imperative?
I know these questions are beyond the scope of a performance review, but inquiring minds would like to know.
What would be much more interesting than "peak responses per second," which is a weird metric to begin with, is the actual histogram of sampled throughputs. Or at the very least a box-and-whisker plot (http://en.wikipedia.org/wiki/Box_plot).
Most folks who have run Rails at scale, for example, find that the untuned garbage collector in MRI (Ruby's default interpreter) introduces a large amount of variance, for example.
Most folks who have run Rails at scale, for example, find that the untuned garbage collector in MRI (Ruby's default interpreter) introduces a large amount of variance, for example.
No love for Flask? I would have love to see how it compare to Django and RoR
I'd love to see how PHP 5.4 compares. In my own app, I saw a noticeable speedup and RAM usage per request dropped by half.
Theres a consistant, considerable gap between their "raw" benchmarks (things like netty, node, plain php, etc.) and frameworks hosted on those same platforms. I think this is something we should keep in mind when we're tuning performance-sensitive portions of APIs and the like. We may actually need to revisit our framework choice and implement selected portions outside of it (just like ruby developers sometimes write performance-critical pieces of gems in C etc.) or optimize the framework further.
I'd like to crunch these numbers further to get a "framework optimization index" which would be the percentage slowdown or ratio of performance between the host platform and the performance of the framework on top of it. I might do this later if I get a chance.