On HTTP Load Testing(mnot.net)
mnot.net
On HTTP Load Testing
http://www.mnot.net/blog/2011/05/18/http_benchmark_rules
2 comments
> Not only that, but the behavior of some cases in hardware VMM layers can cause pretty much complete TLB wipes, thus destroying cache coherency and any memory optimizations the web server takes advantage of.
It is certainly true that VMs can have an impact on webserver performance, but the biggest effects are on performance under saturation, rather than peak performance. In my experience, the mean latency of serving typical dynamic applications from a VM isn't significantly higher on a VM than on similarly powered native hardware. This picture changes at high percentiles (the latency of the top 0.1% of requests, for example) but is certainly not catastrophic for the vast majority of workloads.
Of course, if you are running on a soft-scheduled VM on a busy box, or a very small hard-scheduled slice of hardware, then you are going to get bad performance.
But what concrete difference does this have on the price/performance ratio? It's fairly clear from the experiences of several big web companies that the cost of running web servers on a VM is well worth paying in exchange for the lowered costs of maintenance and system management. A good virtualization infrastructure can significantly reduce whole-system TCO for many workloads.
As for the testing infrastructure, the ability to spin up and down test clusters on services like Amazon EC2 is a huge win. The sizes of the instances (slices/vms/etc) should be chosen to minimize the effect of the client VM, but you don't have to go far to remove the measurement effect of running on a VM in most cases.
It is certainly true that VMs can have an impact on webserver performance, but the biggest effects are on performance under saturation, rather than peak performance. In my experience, the mean latency of serving typical dynamic applications from a VM isn't significantly higher on a VM than on similarly powered native hardware. This picture changes at high percentiles (the latency of the top 0.1% of requests, for example) but is certainly not catastrophic for the vast majority of workloads.
Of course, if you are running on a soft-scheduled VM on a busy box, or a very small hard-scheduled slice of hardware, then you are going to get bad performance.
But what concrete difference does this have on the price/performance ratio? It's fairly clear from the experiences of several big web companies that the cost of running web servers on a VM is well worth paying in exchange for the lowered costs of maintenance and system management. A good virtualization infrastructure can significantly reduce whole-system TCO for many workloads.
As for the testing infrastructure, the ability to spin up and down test clusters on services like Amazon EC2 is a huge win. The sizes of the instances (slices/vms/etc) should be chosen to minimize the effect of the client VM, but you don't have to go far to remove the measurement effect of running on a VM in most cases.
Did you mean to not use a VM for the test client, or did you mean that testing performance of a VM is tricky?
I'm sure he meant testing performance of a VM is tricky, but in actual fact, both are true. Using a VM for the test client is just as susceptible to the issues mentioned as running it as the server. As the article suggests: know your load generator client. This is difficult if it's not consistent.
There are some good nuggets in this article, but he seems to miss a point that I feel is very important and yet is often overlooked in real-world web load testing; emulating an actual client behaviour under different real-world scenarios. Hitting a page or pages X times usually bears almost no resemblance to how a real-world application is used and therefore gives no indication of actual performance and capacity.
To test a web app properly you need the ability to script and/or replay real-world session browsing behaviours, simulating typical interactions. This can be both very time consuming and difficult to set up but yields far more useful and realistic results. In the past I have reasonable success using tools like The Grinder (barely maintained and poorly documented, but very useful and scriptable in Python), and even Microsoft's Web Application Stress tool (scriptable in VB). There are probably better tools than these out there but these at least allow you to code a detailed interaction with you site.
To test a web app properly you need the ability to script and/or replay real-world session browsing behaviours, simulating typical interactions. This can be both very time consuming and difficult to set up but yields far more useful and realistic results. In the past I have reasonable success using tools like The Grinder (barely maintained and poorly documented, but very useful and scriptable in Python), and even Microsoft's Web Application Stress tool (scriptable in VB). There are probably better tools than these out there but these at least allow you to code a detailed interaction with you site.
> There are some good nuggets in this article, but he seems to miss a point that I feel is very important and yet is often overlooked in real-world web load testing; emulating an actual client behaviour under different real-world scenarios.
I don't think he missed that point; he specifically limits his scope to HTTP daemons:
> A lot of people seem to be talking about and performing load tests on HTTP servers, perhaps because there’s a lot more choice of servers these days.
Testing web apps for performance is, as you point out, more difficult. There are tools for taking integration tests and running them in large batches. With some good instrumentation this is a kind of systemic profiling of your app architecture.
But all of his points will still apply.
I don't think he missed that point; he specifically limits his scope to HTTP daemons:
> A lot of people seem to be talking about and performing load tests on HTTP servers, perhaps because there’s a lot more choice of servers these days.
Testing web apps for performance is, as you point out, more difficult. There are tools for taking integration tests and running them in large batches. With some good instrumentation this is a kind of systemic profiling of your app architecture.
But all of his points will still apply.
Perhaps I didn't phrase my comment as well as I should have. Indeed all his points apply and are excellent. As you pointed out, his focus is on testing the HTTP server for performance, rather than the actual apps running on them. The point I was trying to make is that all of the above, _plus_ additionally integration testing and app specific testing should really be done when preparing for the onslaught of a real-world deployment. Admittedly this is deliberately beyond the scope of his article.
Happily, I think we can agree to agree!
The tools built into the more recent versions of Visual Studio (which I believe are different from the Web Application Stress tool) are quite good for modeling accurate load testing. Setting up specific scenarios that are each then segmented into percentages of the overall load test was super straightforward with the test suite tools.
I found that Apache jmeter works well for me. Easier to use than Grinder, but much more useful than the MS tool.
problem with most load testing tools is the fact that you have to script through them. It's a PITA. Especially with PaaS and Heroku/EngineYard/Other-Vendors/-style deployments, time-to-deploy often obsoletes the time-to-script-test-cases. Need something different, agile and continuous.
While it’s tempting to say that the way to achieve this is to run everything on VMs, I’m not convinced that adding another layer of abstraction (as well as more processes running on the host OS) is going to lead to more consistent results. Because of this, dedicated hardware is best. Failing that, just run all of the tests you can in one session, and make it clear that comparisons between different sessions don’t work.
As someone with pretty in-depth knowledge of how a VMM (virtual machine monitor) works on a very low level, using a VM is a truly awful idea. VM exit incidence is pretty much completely unpredictable and can have a huge impact on performance. Not only that, but the behavior of some cases in hardware VMM layers can cause pretty much complete TLB wipes, thus destroying cache coherency and any memory optimizations the web server takes advantage of.
Now, if your goal is measuring performance of web servers on VMMs, go ahead, but be aware that performance consistency is not a plus.