CPU Introspection: Intel Load Port Snooping(gamozolabs.github.io)
gamozolabs.github.io
CPU Introspection: Intel Load Port Snooping
https://gamozolabs.github.io/metrology/2019/12/30/load-port-monitor.html
5 comments
I find it interesting that somebody dedicates his or her life to figuring out how these CPUs work when exactly that information is just laying about in some vault in Santa Clara.
If we're talking about dedicating their life, that's likely to be Agner Fog: https://www.agner.org/optimize/
He's put out the most detailed third-party documentation on Intel and AMD processors that I've ever seen.
He's put out the most detailed third-party documentation on Intel and AMD processors that I've ever seen.
I think I'll disable hyperthreading [0]. :-)
This interesting novel technique [1] can provide an unique window inside the CPU core black box, helping us to better understand how the CPU works internally when it comes to otherwise invisible loads and stores.
While I can't think of any scenario immediately, intuitively I feel this could be useful for those of us seeking to squeeze everything out of a system.
Not the particular case about low level TLB miss / page walk mechanics (we already know TLB misses are bad), but perhaps there are other situations where existing performance counters don't provide as detailed information.
[0]: Yes, I'm aware INVPLG (Invalidate TLB Entries) used in this blog post is a privileged ring 0 instruction. But there's clearly a leak regardless.
[1]: Using a core hyperthread to "spy" other hyperthread on same core.
This interesting novel technique [1] can provide an unique window inside the CPU core black box, helping us to better understand how the CPU works internally when it comes to otherwise invisible loads and stores.
While I can't think of any scenario immediately, intuitively I feel this could be useful for those of us seeking to squeeze everything out of a system.
Not the particular case about low level TLB miss / page walk mechanics (we already know TLB misses are bad), but perhaps there are other situations where existing performance counters don't provide as detailed information.
[0]: Yes, I'm aware INVPLG (Invalidate TLB Entries) used in this blog post is a privileged ring 0 instruction. But there's clearly a leak regardless.
[1]: Using a core hyperthread to "spy" other hyperthread on same core.
Hehe, hyperthreading has some issues. This issue technically works single thread, but it's hard for sensitive data to survive a context switch. That being said, this issue is mitigated in all common OSes and latest microcode.
I'll be curious as to what there is to learn from this. It's more of a longshot goal for me to learn how things work, develop accurate uarch models, and then learn from those models better than I could guess and check hardware results.
Hard to say if it'll go well....
I'll be curious as to what there is to learn from this. It's more of a longshot goal for me to learn how things work, develop accurate uarch models, and then learn from those models better than I could guess and check hardware results.
Hard to say if it'll go well....
> That being said, this issue is mitigated in all common OSes and latest microcode.
Emphasis on this issue, eh?
> I'll be curious as to what there is to learn from this...
I'm also very interested to see what you and the community can discover using this trick!
Emphasis on this issue, eh?
> I'll be curious as to what there is to learn from this...
I'm also very interested to see what you and the community can discover using this trick!
I'll eat my hat for this, but effectively the mitigation to this is clearing all caches and internal buffers in the CPU on each context switch.
I'm sure we'll see more types of leaks, but unless they're actively fetching invalid data [1], there isn't much sensitive data to leak anymore.
I don't think there is much during speculation that can load _new_ data during that window.
[1]: So far almost every CPU bug has leaked something in an internal cache.
I'm sure we'll see more types of leaks, but unless they're actively fetching invalid data [1], there isn't much sensitive data to leak anymore.
I don't think there is much during speculation that can load _new_ data during that window.
[1]: So far almost every CPU bug has leaked something in an internal cache.
A machine clear clears the pipeline. Does it clear these internal caches? There is, of course, no machine clear instruction. Could you construct a machine clearing sequence, insert it into the context switch code and test your hypothesis?
The `verrw` legacy instruction has been added to with microcode to flush internal caches (load buffers, store buffers, etc). Any serializing instruction should (hopefully) cause a pipeline flush. This is the mitigation solution Intel made available to OS developers and should be what is being used.
> I'll eat my hat for this, but effectively the mitigation to this is clearing all caches and internal buffers in the CPU on each context switch.
Are you talking about software context switches or hardware (hyperthreading) ones?
If HW threading constantly clears caches wouldn't it cause a huge performance loss? Isn't that something that can occur 1-100 million times per second?
Sadly, software context switch cache clearing is pretty much given these days.
Are you talking about software context switches or hardware (hyperthreading) ones?
If HW threading constantly clears caches wouldn't it cause a huge performance loss? Isn't that something that can occur 1-100 million times per second?
Sadly, software context switch cache clearing is pretty much given these days.
In this case a privilege transition requires flushing caches. The scheduler has to be aware to not schedule two different permission levels/domains on the same core. It's a huge amount of osdev work to make hyperthreading "safe". I'll be curious if Intel doubles down on HT again.
> I think I'll disable hyperthreading
It will not help you. Anything short of specially built CPU architecture and an OS is useless against hardware level "attacks," if you can even call them that.
CPUs are simply not built with an idea that you have to protect one process from another, and do it on that level of sophistication. You normally don't have that concern if the only person who can run code on a CPU is its user.
But the whole paradigm breaks when you have multiple "tenants" in a single systems, and whose entire setup is not managed by the host.
The same comes when you allow random untrusted code be JITed or even ran as is with WASM.
The only solution against that is to stop people from using "virtual" hosting, and remove JIT compilation of untrusted code.
It will not help you. Anything short of specially built CPU architecture and an OS is useless against hardware level "attacks," if you can even call them that.
CPUs are simply not built with an idea that you have to protect one process from another, and do it on that level of sophistication. You normally don't have that concern if the only person who can run code on a CPU is its user.
But the whole paradigm breaks when you have multiple "tenants" in a single systems, and whose entire setup is not managed by the host.
The same comes when you allow random untrusted code be JITed or even ran as is with WASM.
The only solution against that is to stop people from using "virtual" hosting, and remove JIT compilation of untrusted code.
I hope (perhaps naively) that CPU vendors will wake up, model all side channel data flows and prevent information leakage or provide features for software to use to do so.
CPU features like software controlled cache compartmentalization could also help, preventing cache related side channel leaks.
> The same comes when you allow random untrusted code be JITed or even ran as is with WASM.
If it's single threaded and WASM "syscall" interface doesn't provide anything that can be used as a clock, I don't see there's much to exploit — as long as each syscall interface itself is secure against direct and side channel attacks.
> ...remove JIT compilation of untrusted code.
You could probably still be able to exploit these issues, just somewhat slower. Also interpreters need to access memory, and those patterns are highly predictable.
CPU features like software controlled cache compartmentalization could also help, preventing cache related side channel leaks.
> The same comes when you allow random untrusted code be JITed or even ran as is with WASM.
If it's single threaded and WASM "syscall" interface doesn't provide anything that can be used as a clock, I don't see there's much to exploit — as long as each syscall interface itself is secure against direct and side channel attacks.
> ...remove JIT compilation of untrusted code.
You could probably still be able to exploit these issues, just somewhat slower. Also interpreters need to access memory, and those patterns are highly predictable.
Waking up and realizing the holes need to be plugged is only the first step on the very long path to actually not having any holes.
As evidenced by the struggles of browser developers, making a solid sandbox is -hard-. They've been trying to make JS secure and sidechannel free for a good part of 15 years now, and there are still issues found every quarter.
As evidenced by the struggles of browser developers, making a solid sandbox is -hard-. They've been trying to make JS secure and sidechannel free for a good part of 15 years now, and there are still issues found every quarter.
I read this. It's quite interesting, but I think it is short of a few clear basic definitions of terms.
Could anyone help explain, in a few sentences, what is a Load Port, and why is it interesting in this context?
It appears to be some type of indicator of the proportional time slice given to certain opaque internal processes which are not normally visible to users.
Could anyone help explain, in a few sentences, what is a Load Port, and why is it interesting in this context?
It appears to be some type of indicator of the proportional time slice given to certain opaque internal processes which are not normally visible to users.
CPUs issue instructions through a few ports, each of which services a set of instruction types (eg, arithmetic, memory load/store, vector instructions, etc.). For example here are Skylake’s ports: https://en.wikichip.org/wiki/intel/microarchitectures/skylak...
So getting a trace from the load ports is basically a trace of all memory accesses in the system. Something particularly cool about this work is that you can even see loads that are hidden from software, like a hardware page table walk.
So getting a trace from the load ports is basically a trace of all memory accesses in the system. Something particularly cool about this work is that you can even see loads that are hidden from software, like a hardware page table walk.
Sorry about that. This blog kinda just hopped into the meat of it as I was trying to keep it short. It's largely a followup to a previous blog of mine https://gamozolabs.github.io/metrology/2019/08/19/sushi_roll... where I go into the details and descriptions.
I don’t understand why intel and amd can’t give us access to the cpu cache the same way NVidia does with CUDA for local and shared memory on the gpu. I just don’t buy the hand waiving that the cpu can magically do branch prediction better than a programmer with a static c compiler who actually know what they want in the future. maybe if they had offered cache control on the itanium or phi they wouldn’t have had to cancel them despite the need for reprogramming user software which didn’t stop CUDA.
I agree that we should be able to tell the processor which branch is more likely. Even something as simple as a flag to select between "I prefer you take any jump you encounter" and "I prefer you skip all jumps in this macroblock for speculation purposes" (and also "try to predict smartly" to stick to how it's working now, aka what this article is about). This would give a determined programmer everything he needs to make sure his program is executed optimally.
Back in Pentium 4 days you could use the DS and CS override prefixes on conditional branches to hint taken and not taken, respectively.
Kinda neat, but it's not a thing anymore.
Some more info here: https://stackoverflow.com/questions/14332848/intel-x86-0x2e-...
Kinda neat, but it's not a thing anymore.
Some more info here: https://stackoverflow.com/questions/14332848/intel-x86-0x2e-...