You're missing something very significant: Compilation jobs involve a lot of forking. Forking, memory accesses, and TLB differences across various VM architectures can have an enormous impact on performance.
For example, we found two orders of magnitude performance difference between PV and HVM on ec2 on fork-heavy CI jobs. A difference between 3 and 30 minutes. This difference is trivially reproducible by measuring the time taken to complete a fork() syscall. As the memory size of the parent process grows, the difference becomes more pronounced.
Here's actual data I collected from c3.large instances, using a contrived Ruby one-liner to benchmark fork():
The semantics for write(2) and mmap(2) differ. Neither can be used to implement the other in full.
It's possible to have access patterns which don't demonstrate the efficiencies of mmap but it would be incorrect to suggest they do not exist. The overhead of making a syscall to access an area of a file and copying the data being accessed is significant for many workloads.
Have you tried disabling only defrag? The performance issues I'm familiar with are generally related to defrag running at allocation time. I would be curious to see your results while enabling THP but:
echo never > /sys/kernel/mm/transparent_hugepage/defrag
Ah, thanks for your feedback. I'm never sure how much detail I should dive into. I will add more next time. I'll answer them here just for practice :)
Recall, this is a problem in the 99th percentile. That is, on average our requests are still taking around 20ms to complete -- but one in every hundred or so takes about 200ms.
The first strace output shows a summary table of time, including both our fast 20ms queries and the 1% of slow 200ms queries. In the summary view I am looking not simply at the largest number in the summary but rather indications that my problematic system is spending time doing something that my healthy system is not. Time spent is a zero-sum equation -- the app is either doing what it's supposed to, or not. futex() is where it should be spending time while idle. fsync() time doubles proportionally, which means it's taking time away from the otherwise productive (or, idle) operations. Which means it's potentially the problem.
When I trace the fsync call individually I see a time delta roughly on par with my per-request delay for the slow 1% of queries. Which indicates to me that this call is happening during the slow queries, accounting for the delay in its entirety.
Sort of. The catch is that even a very small write, say just a few megabytes, can drastically change the cost of an fsync(). On my test aws VM even writing just 4 megabytes one time is enough to trigger the problem. Even on an otherwise fully isolated system a few megs may be written from time to time, for example by a management agent like chef or puppet. Or by an application deploy copying out new binaries.
Not recently, no. I do know that a journaled filesystem can exacerbate this sort of problem as it can make extra work. For example: http://lwn.net/Articles/328363/
In a few cases in the past when dealing with unimportant data I have downgraded to ext2 for a nice performance bump.
With fsync() taken out of the equation this is essentially how a naive logger operates. The linux page cache plays the role of shared memory and only under rather heavy contention will a write() incur latency.
The trouble outlined in my blog post is that the logging framework was calling fsync() -- that is, specifically asking to flush the page cache all the way to disk.
Even on the problematic host we only saw this latency issue in the 99th percentile. That is: even on the problem host 99 out of 100 queries were served as expected and only 1 out of 100 saw this additional latency.
Hi, post author here. I didn't make it as clear as I could have, but the difference is that the problematic system had an unrelated process creating slightly more writes. I sort-of glossed over this with the 5%-20% difference in i/o util.
Unrelated write activity on a filesystem can cause cause fsync() calls in any other process to vary wildly in latency. This can be replicated, here's an experiment for you. First, run this:
For example, we found two orders of magnitude performance difference between PV and HVM on ec2 on fork-heavy CI jobs. A difference between 3 and 30 minutes. This difference is trivially reproducible by measuring the time taken to complete a fork() syscall. As the memory size of the parent process grows, the difference becomes more pronounced.
Here's actual data I collected from c3.large instances, using a contrived Ruby one-liner to benchmark fork():
PV: [ec2-user@PV ~]$ ruby -e'$ref=[]; (24..29).each {|n| $ref << "X" * (2 n); start = Time.now.to_f; fork.nil? and exit!; puts Time.now.to_f - start; sleep 1}'
0.00550532341003418
0.011992692947387695
0.02542281150817871
0.05213499069213867
0.10562920570373535
0.21283364295959473
HVM: [ec2-user@HVM ~]$ ruby -e'$ref=[]; (24..29).each {|n| $ref << "X" * (2 n); start = Time.now.to_f; fork.nil? and exit!; puts Time.now.to_f - start; sleep 1}'
0.0005323886871337891
0.0007219314575195312
0.0012335777282714844
0.002183198928833008
0.004210233688354492
0.008102178573608398
As you can see, the difference is considerable. This is all on Amazon ec2, the only difference between these VMs is PV versus HVM.
(edit: HN formatting seems to be eating the exponent operator in the "2 to the n" expression. It's a double *)