Making QUIC Quicker with NIC Offload(dl.acm.org)
dl.acm.org
Making QUIC Quicker with NIC Offload
https://dl.acm.org/doi/abs/10.1145/3405796.3405827
4 comments
>Lesson #1: Data copy between user and kernel space costs around
50% of total CPU usage. This can be avoided by using kernel-bypass
techniques as adopted in Quant.
>Lesson #2: In the presence of a kernel-bypass optimization, crypto operations become the new most expensive operation, requiring up to 40% of CPU resources per connection.
I suppose they did an equivalent optimization ? I have no experience with io_uring, but I assume the gains are similar to kernel bypass optimizations.
>Lesson #2: In the presence of a kernel-bypass optimization, crypto operations become the new most expensive operation, requiring up to 40% of CPU resources per connection.
I suppose they did an equivalent optimization ? I have no experience with io_uring, but I assume the gains are similar to kernel bypass optimizations.
I believe they are talking about "user space networking", like Intel's DPDK. Skipping the kernel and context switches, etc.
This gets reinvented as a system matures, I think.
IBM mainframes had TCP offload in the early 90's at least. (The NIC in that case was a PC running PS/2 plus some routing software. Worked great.)
IBM mainframes had TCP offload in the early 90's at least. (The NIC in that case was a PC running PS/2 plus some routing software. Worked great.)
and they surely weren't the first to do that either, I assume now.
FYI, it's possible to edit comments to add further thoughts. :)
the world should be append only!
Have we addressed the privacy issues with QUIC?
What do you mean by privacy issues with QUIC? As I understand it, it’s just another application layer protocol.
Iirc, Brave browser began disabling QUIC because the connected server was able to fingerprint the connecting party, and they deemed it an intentional part of the protocol design. Essentially connecting anonymously to a server is not possible. I'll try and dig up the blog post.
Hmm, researchers with the 'National University of Defense Technology' propound on the benefits of offloading encryption/decryption to a separate processor/FPGA on the Network Interface Card. Just for speed, I'm sure!
This has been done for years, eg with Chelsio NICs, and indeed it can speed things up quite a bit.
Wiretapping has been done for years, yes. NOBUS encryption compromise has a long history, and speeds things up a lot. (for the NSA)
How do you imagine this wiretapping works? This NIC is installed in your servers, and is fed AEAD keys that are derived from key exchange with a client by your TLS stack (and so private keys exist on the host, not on the NIC). This allows the NIC to decrypt/encrypt flows that pass through it and free up CPU cycles. QUIC requires forward secrecy for key exchange, so every flow will use a different AEAD key already, meaning any snooped key can only be used to decrypt the current flow, not any others. Every modern offload NIC uses this basic design, more or less. Where's the wiretap? Is the NIC going to somehow store every intermediate AEAD keys and escrow it to the NSA somehow? What does NOBUS have to do with any of this? And why wouldn't they just backdoor the motherboard/OS/CPU itself to acquire private keys directly?
If a special packet from the network -- maybe even just an innocuous-looking magic string in plaintext! – ran some undisclosed software on the NIC, how would you know?
If that software then exfiltrated valuable info – session keys, or a digest of hotwords seen, or every string typed into targeted HTML form fields – how would you know? Ask the NIC for a record of all its communication? Do you own separate forensic decryption of every session with every possible-surveillance host it's ever connected to? Monitor the local RF spectrum for intentional leakage to other nearby compromised devices?
I think attempted backdoors in motherboards, OS, and CPU are worth considering, too! Why not all of the above, if the budget & cleverness of your staff allows it?
But proprietary NIC firmware/chips are an especially arcane, independent, essentially-network-connected part of the system. It's a per-machine 'high ground' for observing all traffic, especially if entrusted with all network encryption/decryption, much like how routers and telecom switching centers have also been especially interesting to surveillance hackers.
If that software then exfiltrated valuable info – session keys, or a digest of hotwords seen, or every string typed into targeted HTML form fields – how would you know? Ask the NIC for a record of all its communication? Do you own separate forensic decryption of every session with every possible-surveillance host it's ever connected to? Monitor the local RF spectrum for intentional leakage to other nearby compromised devices?
I think attempted backdoors in motherboards, OS, and CPU are worth considering, too! Why not all of the above, if the budget & cleverness of your staff allows it?
But proprietary NIC firmware/chips are an especially arcane, independent, essentially-network-connected part of the system. It's a per-machine 'high ground' for observing all traffic, especially if entrusted with all network encryption/decryption, much like how routers and telecom switching centers have also been especially interesting to surveillance hackers.
The thing is, all your said, applies to all NICs we used these days. There are nothing special in these NICs with advanced offload features.
Yes, it's an attractive place to add surveillance code for all NICs. But not all NICs take over encryption/decryption. And if other components do the encryption/decryption, NICs don't see plaintext - just (lots of) metadata.
If/when they take over encryption/decryption functions, that would make them an even better place for hard-to-detect, user-disloyal surveillance code.
If/when they take over encryption/decryption functions, that would make them an even better place for hard-to-detect, user-disloyal surveillance code.
> If that software then exfiltrated valuable info – session keys, or a digest of hotwords seen, or every string typed into targeted HTML form fields – how would you know?
How does this magical exploit work, exactly? How would it extract keywords from forms, when forms are ridiculously dynamic per-website, per user, and per form? (Many forms aren't even actually that, but text boxes that are fed directly by JavaScript event handlers into a Websocket, these days.) How do they target these groups using and installing these the NICs, and ensure they're all put in the right place? Do they literally backdoor every device that exists? No meaningful backdoors were ever found in Supermicro devices, for instance, despite immense pressure to find them. The same with session keys, what "session keys" are being compromised? Session keys exist at the TLS level, the application level, and conceptually many others. TLS keys already aren't useful for anything beyond their used connections for instance, and flow identification can vary in many ways, so how would they correlate user-to-flow in order to target attacks? If they're not doing targeted attacks, then what are they doing?
How is it going to exfiltrate that data if the NIC is on an isolated network (e.g. in some mesh topology, because it's only used intra-rack)? How do they plan for that, by exploiting the software stack? If they can exploit the software stack already, why not just compromise the host? You're telling me NSA spooks can in advance make sure your datacenter rack is organized so that RF signals can be used to wirelessly transport data on the spectrum and nobody has ever had proof of this, but they can't just, you know, write a software exploit? Issue a gag order? And that they use this capability with some regularity? If it wasn't regular, why do it in the first place when the risk of discovery is higher? How do they predict these ridiculously variable conditions in advance, and how are they not discovered? And above all, why would all this be something you immediately jump to, instead of just, say, normal infosec style attacks? You can literally buy 0days for COTS software, on the market. Your original post was motivated by fear of some Chinese person working for the Chinese government. Why? Because China is a totalitarian government but also they somehow need university students to make backdoor'd NICs, in conjunction with researchers from Europe who wouldn't, like, detect them? The ruling party controls the entire military and governs the internet and can already decrypt all connections, but they need this too?
This kind of stuff is just sci-fi/mil-tech fantasy scenarios. We can all hash out incredibly-specific ridiculous infosec attacks all day that we could pull off in a controlled environment, but that doesn't mean any of them are actually more useful or viable than others. People on this site desperately want to believe the world is filled with action movie sci-fi hacker scenarios that can just be defeated in a technical one-up kind of way, because it means the solution is purely a technical problem, rather than the real alternative: you will get thrown in a dark jail cell for 500 years without appeal if you don't comply with this order to install a backdoor directly in your datacenter.
How does this magical exploit work, exactly? How would it extract keywords from forms, when forms are ridiculously dynamic per-website, per user, and per form? (Many forms aren't even actually that, but text boxes that are fed directly by JavaScript event handlers into a Websocket, these days.) How do they target these groups using and installing these the NICs, and ensure they're all put in the right place? Do they literally backdoor every device that exists? No meaningful backdoors were ever found in Supermicro devices, for instance, despite immense pressure to find them. The same with session keys, what "session keys" are being compromised? Session keys exist at the TLS level, the application level, and conceptually many others. TLS keys already aren't useful for anything beyond their used connections for instance, and flow identification can vary in many ways, so how would they correlate user-to-flow in order to target attacks? If they're not doing targeted attacks, then what are they doing?
How is it going to exfiltrate that data if the NIC is on an isolated network (e.g. in some mesh topology, because it's only used intra-rack)? How do they plan for that, by exploiting the software stack? If they can exploit the software stack already, why not just compromise the host? You're telling me NSA spooks can in advance make sure your datacenter rack is organized so that RF signals can be used to wirelessly transport data on the spectrum and nobody has ever had proof of this, but they can't just, you know, write a software exploit? Issue a gag order? And that they use this capability with some regularity? If it wasn't regular, why do it in the first place when the risk of discovery is higher? How do they predict these ridiculously variable conditions in advance, and how are they not discovered? And above all, why would all this be something you immediately jump to, instead of just, say, normal infosec style attacks? You can literally buy 0days for COTS software, on the market. Your original post was motivated by fear of some Chinese person working for the Chinese government. Why? Because China is a totalitarian government but also they somehow need university students to make backdoor'd NICs, in conjunction with researchers from Europe who wouldn't, like, detect them? The ruling party controls the entire military and governs the internet and can already decrypt all connections, but they need this too?
This kind of stuff is just sci-fi/mil-tech fantasy scenarios. We can all hash out incredibly-specific ridiculous infosec attacks all day that we could pull off in a controlled environment, but that doesn't mean any of them are actually more useful or viable than others. People on this site desperately want to believe the world is filled with action movie sci-fi hacker scenarios that can just be defeated in a technical one-up kind of way, because it means the solution is purely a technical problem, rather than the real alternative: you will get thrown in a dark jail cell for 500 years without appeal if you don't comply with this order to install a backdoor directly in your datacenter.
Since NIC is used to encrypt the traffic, it can purposefully have vulnerability that NSA knows about?
Fixing NIC is not as easy as fixing software.
Fixing NIC is not as easy as fixing software.
Software - eg the Linux kernel - can also contain vulnerability only NSA knows about.
In any case, security track record of crypto offload NICs is pretty good.
In any case, security track record of crypto offload NICs is pretty good.
> Software - eg the Linux kernel - can also contain vulnerability only NSA knows about.
But when it is found, it can be easily fixed
> In any case, security track record of crypto offload NICs is pretty good.
Maybe because the only way to see what it does is to use an electron microscope?
But when it is found, it can be easily fixed
> In any case, security track record of crypto offload NICs is pretty good.
Maybe because the only way to see what it does is to use an electron microscope?
To make this vulnerable to third parties not physically in control of the hardware (in which case the PCI bus could just be sniffed for the key, or have it grabbed from RAM). Pretty much the only option is side channels.
They can't change the algorithm or other NICs that don't offload the encryption/decryption wouldn't be able to successfully decrypt it. They couldn't send "extra" packets with the key somewhere else without someone very easily detecting that anomalous traffic not being generated by the system itself.
With QUIC there are almost no-unencrypted fields by design, and those have very firm well defined meanings, so to inject steganographic hidden copies of the key in the packets themselves is infeasible. Maybe you could sneak something out by changing source ports but that's risky, and liable to be detected as well.
So that leaves side channels, which is basically only a timing channel for remote attackers in this case. At the speeds and packet throughput where hardware acceleration of the crypto matters, any buffers in the first switch and/or router that packet hits will remove the precision required for any level of intentional timing attack that could be introduced without detection.
So no it almost certainly couldn't have a vulnerability like that unless the crypto algorithms themselves are broken, in that case this whole discussion is moot as it doesn't matter if its accelerated or not.
> Fixing NIC is not as easy as fixing software
This kind of thing is frequently implemented in FPGAs which can be updated with firmware. The firmware will likely be closed source, but that hasn't stopped people before from fixing firmwares without the original manufacturers knowledge or consent.
It's a lot more work but turns out the people that would be targeted by an attack fall into two categories, those who have the financial ability to pay for a security team to handle these kinds of vulnerabilities in house (or at least mitigate them) and normal people who couldn't bear the brunt of a nation state targeting them even if this one vulnerability didn't exist.
The NSA has been caught doing some seriously shady things we all know that, and no one is going to seriously argue they've stopped trying and developing new things, but it's not even close to reasonable to assume that any paper that has "crypto" present in it is a conspiracy from the NSA or any other intelligence agency.
They can't change the algorithm or other NICs that don't offload the encryption/decryption wouldn't be able to successfully decrypt it. They couldn't send "extra" packets with the key somewhere else without someone very easily detecting that anomalous traffic not being generated by the system itself.
With QUIC there are almost no-unencrypted fields by design, and those have very firm well defined meanings, so to inject steganographic hidden copies of the key in the packets themselves is infeasible. Maybe you could sneak something out by changing source ports but that's risky, and liable to be detected as well.
So that leaves side channels, which is basically only a timing channel for remote attackers in this case. At the speeds and packet throughput where hardware acceleration of the crypto matters, any buffers in the first switch and/or router that packet hits will remove the precision required for any level of intentional timing attack that could be introduced without detection.
So no it almost certainly couldn't have a vulnerability like that unless the crypto algorithms themselves are broken, in that case this whole discussion is moot as it doesn't matter if its accelerated or not.
> Fixing NIC is not as easy as fixing software
This kind of thing is frequently implemented in FPGAs which can be updated with firmware. The firmware will likely be closed source, but that hasn't stopped people before from fixing firmwares without the original manufacturers knowledge or consent.
It's a lot more work but turns out the people that would be targeted by an attack fall into two categories, those who have the financial ability to pay for a security team to handle these kinds of vulnerabilities in house (or at least mitigate them) and normal people who couldn't bear the brunt of a nation state targeting them even if this one vulnerability didn't exist.
The NSA has been caught doing some seriously shady things we all know that, and no one is going to seriously argue they've stopped trying and developing new things, but it's not even close to reasonable to assume that any paper that has "crypto" present in it is a conspiracy from the NSA or any other intelligence agency.
> They couldn't send "extra" packets with the key somewhere else without someone very easily detecting that anomalous traffic not being generated by the system itself.
Why would it matter if the traffic is detected? The companies owning the hardware are usually working with the NSA or compelled to do so, so the extra traffic is expected: https://archive.nytimes.com/www.nytimes.com/interactive/2013...
You can see part of the budget is straight out putting the backdoors in the VPN/web encryption chips design.
Why would it matter if the traffic is detected? The companies owning the hardware are usually working with the NSA or compelled to do so, so the extra traffic is expected: https://archive.nytimes.com/www.nytimes.com/interactive/2013...
You can see part of the budget is straight out putting the backdoors in the VPN/web encryption chips design.
Exactly, if we learned anything from Snowden disclosure, is that in many cases they simply approach the companies and they will implement such functionality.
What's the point of those down voting comments when there is clearly an evidence that those things happened in the past.
What's the point of those down voting comments when there is clearly an evidence that those things happened in the past.
Okay, but I'm asking what does it look like? What kind of vulnerability? They presumably aren't using SuperDuper Secret Wifi to mirror data wirelessly to the moon, right? There are a limited number of outcomes at some point. They can't exactly change the encryption algorithms, otherwise clients fail to connect, and modern TLS (and QUIC) are designed to reduce algorithm agility in the name of preventing downgrade attacks and insecure suites. And they can't just break AES with an alien computer, because if so, why bother with the NIC at all? And again: How do they escrow the data they want out of the network, considering the extremely variable (and potentially secured, unknown, hostile) network conditions? If they can do that with some kind of host exploit or whatever, why not just take private keys in the first place? They could just snip ground cables then and be done with it, which is exactly how they got Google. (The most realistic case I can think of is somehow compromising entropy generation, perhaps.)
And finally, why do any of this when you can almost definitely just issue a gag order to a legal council, or behind-the-door threats to a foreign government agency to tow the line, or any number of things? You're dealing with governments who have immense global influence, not scrappy hackers who only have their wits and old laptops about them.
I'm not saying agencies don't have exploits, or they don't use them, or they don't spy on a lot of data, or that even some backdoors aren't real. But if you're looking a NIC offload device, immediately claim "Wiretapping", and can't actually explain how it wiretaps anything or what the attack model is, it's really just random speculation and fear mongering.
And finally, why do any of this when you can almost definitely just issue a gag order to a legal council, or behind-the-door threats to a foreign government agency to tow the line, or any number of things? You're dealing with governments who have immense global influence, not scrappy hackers who only have their wits and old laptops about them.
I'm not saying agencies don't have exploits, or they don't use them, or they don't spy on a lot of data, or that even some backdoors aren't real. But if you're looking a NIC offload device, immediately claim "Wiretapping", and can't actually explain how it wiretaps anything or what the attack model is, it's really just random speculation and fear mongering.
While I don't think it's likely, it's not hard to conceive a scenario where the NIC purposely weakens the security for attackers in the know.
Purely theoretical (and I'm not a crypto guy, so please do correct me if this is nonsense), but imagine a scheme whereby the IV is chosen to be the first few bytes of the private key xor the port tuple.
This could reduce the difficulty of brute forcing the key, and no extra traffic need be generated - we already know that the NSA operates passive observers, and has even placed such systems inside corporate networks in the past.
EDIT: As to why they'd do this instead of getting a gag order - because they can? Because there's less oversight? Safest to assume that any technical capability will be abused sooner or later.
Purely theoretical (and I'm not a crypto guy, so please do correct me if this is nonsense), but imagine a scheme whereby the IV is chosen to be the first few bytes of the private key xor the port tuple.
This could reduce the difficulty of brute forcing the key, and no extra traffic need be generated - we already know that the NSA operates passive observers, and has even placed such systems inside corporate networks in the past.
EDIT: As to why they'd do this instead of getting a gag order - because they can? Because there's less oversight? Safest to assume that any technical capability will be abused sooner or later.
> but imagine a scheme whereby the IV is chosen to be the first few bytes of the private key xor the port tuple.
Again, the NIC doesn't choose the IV. It is given an IV by the host system, which is derived from key exchange in software, and that IV must match what the other side of the link derives from its own key exchange operation. It has no choice but to use the IV given. Otherwise, the two parties can't communicate. So the NIC would have to attack the host system somehow to engage in this attack, but then it could just steal a private key anyway and get all communications forever. This is basic Diffie-Hellman/TLS 101.
This kind of "I'm not an expert, but let me make up a scenario completely divorced from reality..." thing is what I'm talking about when I say speculation/FUD. It sounds sufficiently "techie smart" to pass a trivial smell test but otherwise instantly falls apart.
> As to why they'd do this instead of getting a gag order - because they can? Because there's less oversight? Safest to assume that any technical capability will be abused sooner or later.
Any person in your life that you know could suddenly commit a horrible crime, just "because they can." Do you think they will? Is that reason to assume they will? "Because they can" ignores a basic aspect of how decisions are made, which is understanding their motivations and reasoning.
And less oversight from what? These gag orders are already enforced in secret courts. Governments exert pressure on each other, behind closed doors and through agreements like trade sanctions, to force other governments to comply. Theres's already "no oversight" in the process, by design it avoids oversight. Spooks can literally walk into your datacenter and pull a rack out of the cage and there's nothing you can do about it unless you want to get thrown in a dark hole for 500 years. Even if they had to resort to techie tricks, why is the scenario you imagine any more plausible than a thousand simpler, alternative options? Multi-million dollar corporations get ransomware'd all the time, and it's not like the culprits need hardware backdoors to do it.
Again: these agencies have exploits, and for a reason. They certainly use them. They have backdoors. That doesn't mean we just get to turn our brains off the instant something we don't understand mildly spooks us and assign complete impossibilities as the culprit. You're not far from just doing high-brow "lizard people control society" stuff at that point.
Again, the NIC doesn't choose the IV. It is given an IV by the host system, which is derived from key exchange in software, and that IV must match what the other side of the link derives from its own key exchange operation. It has no choice but to use the IV given. Otherwise, the two parties can't communicate. So the NIC would have to attack the host system somehow to engage in this attack, but then it could just steal a private key anyway and get all communications forever. This is basic Diffie-Hellman/TLS 101.
This kind of "I'm not an expert, but let me make up a scenario completely divorced from reality..." thing is what I'm talking about when I say speculation/FUD. It sounds sufficiently "techie smart" to pass a trivial smell test but otherwise instantly falls apart.
> As to why they'd do this instead of getting a gag order - because they can? Because there's less oversight? Safest to assume that any technical capability will be abused sooner or later.
Any person in your life that you know could suddenly commit a horrible crime, just "because they can." Do you think they will? Is that reason to assume they will? "Because they can" ignores a basic aspect of how decisions are made, which is understanding their motivations and reasoning.
And less oversight from what? These gag orders are already enforced in secret courts. Governments exert pressure on each other, behind closed doors and through agreements like trade sanctions, to force other governments to comply. Theres's already "no oversight" in the process, by design it avoids oversight. Spooks can literally walk into your datacenter and pull a rack out of the cage and there's nothing you can do about it unless you want to get thrown in a dark hole for 500 years. Even if they had to resort to techie tricks, why is the scenario you imagine any more plausible than a thousand simpler, alternative options? Multi-million dollar corporations get ransomware'd all the time, and it's not like the culprits need hardware backdoors to do it.
Again: these agencies have exploits, and for a reason. They certainly use them. They have backdoors. That doesn't mean we just get to turn our brains off the instant something we don't understand mildly spooks us and assign complete impossibilities as the culprit. You're not far from just doing high-brow "lizard people control society" stuff at that point.
Assuming they're talking about Linux here, I wonder if they used io_uring.