CPU utilization is wrong [video](opensource.com)
opensource.com
CPU utilization is wrong [video]
https://opensource.com/article/18/4/cpu-utilization-wrong
6 comments
This nigh-useless blurb links to a Youtube video: https://youtu.be/QkcBASKLyeU
Yeah, can someone quick summarize this? Can't watch where I am.
edit: I found his blog post on this very topic. This should be the source: http://www.brendangregg.com/blog/2017-05-09/cpu-utilization-...
Also, now that we have the blog link, we have some previous HN threads: https://news.ycombinator.com/item?id=14301739
edit: I found his blog post on this very topic. This should be the source: http://www.brendangregg.com/blog/2017-05-09/cpu-utilization-...
Also, now that we have the blog link, we have some previous HN threads: https://news.ycombinator.com/item?id=14301739
Brendan Gregg diagnoses a performance problem on Netflix servers. He starts with CPU Utilization, but CPU Utilization metrics from top or ps don't give enough detail. He checks out more detailed performance views (including flame graphs, and some custom tools he wrote himself) to eventually narrow the problem down to the TLB: Translation-Lookaside Buffer. The mechanism the CPU uses to implement virtual memory.
Finally, he realizes that TLB-Misses were caused by Meltdown / Spectre Patches. Which is a difference between the two servers he was diagnosing the performance problem for.
Finally, he realizes that TLB-Misses were caused by Meltdown / Spectre Patches. Which is a difference between the two servers he was diagnosing the performance problem for.
Beat you to it but thanks :P
To save you the time of watching the video: Spectre / Meltdown mitigations (page table isolation) cause execution inefficiency because of TLB cache misses.
So first of all: the lightning talk is certainly "worth it". Brendan Gregg is very knowledgeable and I can certainly recommend watching the talk. Its a good overview into the mindset of low-level optimization.
However, the core premise, the "headline" if you will of this talk, is utterly wrong. I get it: the headline needs to be a bit provocative to draw eyes, but its closer to "clickbait" than a good title of the talk.
The TL;DR of the talk is: Spectre/Meltdown caused a slowdown. But said slowdown is coming up as a ~+20% "CPU Utilization" under Linux performance measurement (ie: top or ps). This slowdown is because of a severe increase in TLB-misses.
Which... doesn't seem "wrong" to me. If the CPU is stalling because of TLB-buffer misses (kinda similar to an L2 cache miss or whatnot), then that's certainly CPU-time that is being wasted on something. Perhaps it isn't using power since the CPU is fully stalled, but the CPU is certainly "utilized" during that timeframe. Aside from hyperthreading: you can't really do something during the stall.
----------
Anyway, my criticism aside, the talk is certainly worthwhile to watch. And that's the important bit. So go ahead and watch the talk, even if I have a few pedantic qualms about the phrasing of the headline.
The real benefit of the talk is how Brendan Gregg diagnosed this performance problem, which eventually led to his conclusion about TLB-misses + Spectre. Demonstrating this skill and walking through a system to be able to draw this kind of conclusion is incredibly important.
I just don't... really like the title of the talk I guess. A rather minor concern all else considered.
However, the core premise, the "headline" if you will of this talk, is utterly wrong. I get it: the headline needs to be a bit provocative to draw eyes, but its closer to "clickbait" than a good title of the talk.
The TL;DR of the talk is: Spectre/Meltdown caused a slowdown. But said slowdown is coming up as a ~+20% "CPU Utilization" under Linux performance measurement (ie: top or ps). This slowdown is because of a severe increase in TLB-misses.
Which... doesn't seem "wrong" to me. If the CPU is stalling because of TLB-buffer misses (kinda similar to an L2 cache miss or whatnot), then that's certainly CPU-time that is being wasted on something. Perhaps it isn't using power since the CPU is fully stalled, but the CPU is certainly "utilized" during that timeframe. Aside from hyperthreading: you can't really do something during the stall.
----------
Anyway, my criticism aside, the talk is certainly worthwhile to watch. And that's the important bit. So go ahead and watch the talk, even if I have a few pedantic qualms about the phrasing of the headline.
The real benefit of the talk is how Brendan Gregg diagnosed this performance problem, which eventually led to his conclusion about TLB-misses + Spectre. Demonstrating this skill and walking through a system to be able to draw this kind of conclusion is incredibly important.
I just don't... really like the title of the talk I guess. A rather minor concern all else considered.
I think the problem Brendan is pointing out is that if we look at a task that appears CPU-bound, we might think that one solution to make things go faster is just make the CPU go faster. But beefing up the CPU won't help, because it'll just spend the same amount of time waiting for e.g. the TLB to make the round trip lookup.
I think this is really a discussion about the Von Neumann Bottleneck (VNBN). The slowdown here is caused by the VNBN, but %CPU utilization is "disguising" slowdowns caused by this bottleneck as a CPU bottleneck.
The VNBN is the primary reason Intel spends so much of their die space on caches -- it's really holding us back. Intel is basically a glorified SRAM vendor, selling vast blocks of memory attached as closely to a processor as possible, because it's just so damn expensive to fetch it from anywhere else, thanks to the VNBN.
> Aside from hyperthreading: you can't really do something during the stall.
This is true, but hyperthreading doesn't truly help us because the unit of concurrency -- the thread here -- is too coarse, and everyone is competing over the same resource: memory bandwidth (again, thanks VNBN), so any time you spend on another task is going to increase return time of the original task you were waiting for.
But we can do something here --- use a different architecture. Imagine a dataflow machine, which sidesteps the VNBN entirely. Now if your processor is stalled waiting for a TLB return, there’s a vast pile of work that can be freely executed. Sure we have other resources at play now (like whatever message passing bandwidth we have), but we no longer have to worry about the VNBN.
I think this is really a discussion about the Von Neumann Bottleneck (VNBN). The slowdown here is caused by the VNBN, but %CPU utilization is "disguising" slowdowns caused by this bottleneck as a CPU bottleneck.
The VNBN is the primary reason Intel spends so much of their die space on caches -- it's really holding us back. Intel is basically a glorified SRAM vendor, selling vast blocks of memory attached as closely to a processor as possible, because it's just so damn expensive to fetch it from anywhere else, thanks to the VNBN.
> Aside from hyperthreading: you can't really do something during the stall.
This is true, but hyperthreading doesn't truly help us because the unit of concurrency -- the thread here -- is too coarse, and everyone is competing over the same resource: memory bandwidth (again, thanks VNBN), so any time you spend on another task is going to increase return time of the original task you were waiting for.
But we can do something here --- use a different architecture. Imagine a dataflow machine, which sidesteps the VNBN entirely. Now if your processor is stalled waiting for a TLB return, there’s a vast pile of work that can be freely executed. Sure we have other resources at play now (like whatever message passing bandwidth we have), but we no longer have to worry about the VNBN.
Well, this particular slowdown was caused by the Spectre / Meltdown patch.
In particular, the Linux KPTI (https://en.wikipedia.org/wiki/Kernel_page-table_isolation) which prevents Meltdown from affecting the Kernel. The hardware fix is simple: buy AMD Epyc instead. At least, in the nearterm. Intel's "Icelake" system will have the hardware fix so that KPTI can die sometime in 2019. Its not normal to flush the TLB on every systemcall. But its a necessary precaution as long as the CPU is vulnerable to Meltdown.
In particular, the Linux KPTI (https://en.wikipedia.org/wiki/Kernel_page-table_isolation) which prevents Meltdown from affecting the Kernel. The hardware fix is simple: buy AMD Epyc instead. At least, in the nearterm. Intel's "Icelake" system will have the hardware fix so that KPTI can die sometime in 2019. Its not normal to flush the TLB on every systemcall. But its a necessary precaution as long as the CPU is vulnerable to Meltdown.
There's also the option of not running KPTI at all... database servers are the most heavily impacted, and since they only run trusted code there's not much point in patching them.
Hmmm... I don't like the idea of disabling security for performance.
I'm not an expert on this, but I'd turn on Huge Tables (1GB) first. Big databases / datasets across RAM can be manipulated in 1GB chunks by the OS instead of 4kB (!!!) chunks. The Skylake CPU can only cache ~4 Huge Tables, but an additional set of Large Tables (2MB) should also help performance. If properly configured of course.
I would hope that those changes would mitigate most of the KPTI issues with regards to databases. But I really don't know.
I'm not an expert on this, but I'd turn on Huge Tables (1GB) first. Big databases / datasets across RAM can be manipulated in 1GB chunks by the OS instead of 4kB (!!!) chunks. The Skylake CPU can only cache ~4 Huge Tables, but an additional set of Large Tables (2MB) should also help performance. If properly configured of course.
I would hope that those changes would mitigate most of the KPTI issues with regards to databases. But I really don't know.
Do you mean Huge Pages?
Would the mill architecture avoid this bottleneck?
https://millcomputing.com
At a very brief glance I don't see anything to suggest they would.
So is the CPU available to do something else or not?
The particular thread that is waiting for data is not, but in a SMT system the other thread has access to the full execution resources of the CPU.
That's great and I agree with him. We have to pay attention to the stale.
arseraptor(1)