Netflix HTTPS performance experiments for large scale content distribution(lists.w3.org)
lists.w3.org
Netflix HTTPS performance experiments for large scale content distribution
https://lists.w3.org/Archives/Public/www-tag/2015Apr/0027.html
32 comments
If you're just pushing raw bytes down a socket, you can use sendfile(2) to direct cached data directly to the NIC without copying it to a private buffer first. This is a pretty big deal at 40Gbps, and Netflix have funded quite a bit of work towards improving it:
https://secure.freshbsd.org/search?project=freebsd&q=netflix+sendfile
It's surprising. They say that they are using the hardware AES support in the CPU too. They do mention that they are no longer able to take advantage of sendfile() which was letting the bulk of the data transfer to happen completely in kernel space.
So, in their case, they had already optimized things to such a degree that they were having a lot of throughput and very little CPU usage per byte. In fact, it's possible that on their old system the data streams wouldn't have to be touched a byte at a time, as they could be loaded into main memory via DMA from the disk and once that finished they could be transferred to the network card, again via DMA. Once sendfile() is called, no further context switches are necessary for that request.
Testing with `openssl speed -evp AES128` on an i5-4670K I'm able to get 6.7 gigabits of encryption out of AES-NI. I tried running two concurrent copies and they each got that speed, which tells me that it's per-core. That's easily enough speed to saturate a 10G connection.
It's quite possible that they have much older or slower hardware involved, since when you're building a server like this normally the network or disks are the limiting factor.
So, in their case, they had already optimized things to such a degree that they were having a lot of throughput and very little CPU usage per byte. In fact, it's possible that on their old system the data streams wouldn't have to be touched a byte at a time, as they could be loaded into main memory via DMA from the disk and once that finished they could be transferred to the network card, again via DMA. Once sendfile() is called, no further context switches are necessary for that request.
Testing with `openssl speed -evp AES128` on an i5-4670K I'm able to get 6.7 gigabits of encryption out of AES-NI. I tried running two concurrent copies and they each got that speed, which tells me that it's per-core. That's easily enough speed to saturate a 10G connection.
It's quite possible that they have much older or slower hardware involved, since when you're building a server like this normally the network or disks are the limiting factor.
> Testing with `openssl speed -evp AES128` on an i5-4670K I'm able to get 6.7 gigabits of encryption out of AES-NI
Bit of a crappy test, though - tight loop over a single 8k buffer (used in-place). It probably executes entirely in L1/L2.
They provide details of the hardware they're using: https://openconnect.itp.netflix.com/hardware/
The fanciest in their "IO-optimized" SSD-driven beast is a 2.7GHz 12 core Ivy Bridge. You've got a 20% clock advantage, and Haswell reportedly reduced AES-NI clock latency from 8 to 7 cycles for a ~14% improvement. Doing the math I get 55Gbps for the lot.
That's without doing any packet processing, no waiting on memory accesses, no interrupts, no context switching, just perfect scaling with a tight loop over the same 8k buffer - and they're driving 4 10Gbps NICs. Not much left over.
Bit of a crappy test, though - tight loop over a single 8k buffer (used in-place). It probably executes entirely in L1/L2.
They provide details of the hardware they're using: https://openconnect.itp.netflix.com/hardware/
The fanciest in their "IO-optimized" SSD-driven beast is a 2.7GHz 12 core Ivy Bridge. You've got a 20% clock advantage, and Haswell reportedly reduced AES-NI clock latency from 8 to 7 cycles for a ~14% improvement. Doing the math I get 55Gbps for the lot.
That's without doing any packet processing, no waiting on memory accesses, no interrupts, no context switching, just perfect scaling with a tight loop over the same 8k buffer - and they're driving 4 10Gbps NICs. Not much left over.
>> It probably executes entirely in L1/L2.
L1!
L1!
Gmail servers probably do actual work. Netflix's CDN just copies bytes from disk to network, so TLS overhead is a greater percentage.
I'd also the content type has something to do with it. Encrypting a 10kb email is a lot faster than encrypting an 3GB movie.
Isn't it just compressed "on the stream"? It's a matter of throughput, not size of individual files.
They are completely different use cases though, so it shouldn't be too surprising that the results are different.
Gmail: Lots of small requests which each last a short time.
Netflix: Small number of large requests with connections that stay open a long time (possibly hours).
Gmail: Lots of small requests which each last a short time.
Netflix: Small number of large requests with connections that stay open a long time (possibly hours).
I understand the need for HTTPS when browsing Netflix, but is there really any benefit in encrypting the actual media stream?
The file is already encrypted for DRM purposes... if thry want to go beyond that, how about a system where each edge server gets a differently encoded file from the master?
The file is already encrypted for DRM purposes... if thry want to go beyond that, how about a system where each edge server gets a differently encoded file from the master?
Without encrypting the video contents, anyone sniffing the data stream can easily tell what you're watching, when you paused, or skipped, or re-played, etc. They may not be able to steal your money with that data, but it still exposes real, personal information.
(though... I was just thinking about this, and due to the use of variable-bitrate encoding, the varied rate of data transmission over time (even though encrypted) would give each video a fingerprint that an observer could use to determine whatever it is that you're watching, anyway, albeit with greater effort that simply inspecting the unencrypted stream)
(though... I was just thinking about this, and due to the use of variable-bitrate encoding, the varied rate of data transmission over time (even though encrypted) would give each video a fingerprint that an observer could use to determine whatever it is that you're watching, anyway, albeit with greater effort that simply inspecting the unencrypted stream)
If the DRM key is transferred securely, how can an attacker tell what you're watching without fingerprinting the entire Netflix library? You can't just open up parts of the stream and parse it as a plain video file to my understanding.
I'll give you the rewind scenario, but play/pause events are going to be equally obvious in HTTPS. Besides, what exactly does the attacker learn? That you were watching Netflix during that time? The HTTPS traffic to Netflix will be an equally good indicator of that.
I'll give you the rewind scenario, but play/pause events are going to be equally obvious in HTTPS. Besides, what exactly does the attacker learn? That you were watching Netflix during that time? The HTTPS traffic to Netflix will be an equally good indicator of that.
Fingerprinting the entire Netflix library - especially if you limit yourself to popular shows - probably isn't that hard. I concur with not really understanding the point, though.
One possibly relevant factor is that shared encryption keys make it harder to authenticate the data: you can't rely on an HMAC, and RSA verifying every frame would be too slow. You can still do it efficiently with a hash tree, but I bet Netflix currently isn't, and TLS allows browsers to enforce verification on behalf of the user. Without verification, you could potentially exploit the browser with malformed audio or video data, or freak the user out by injecting some other video...
One possibly relevant factor is that shared encryption keys make it harder to authenticate the data: you can't rely on an HMAC, and RSA verifying every frame would be too slow. You can still do it efficiently with a hash tree, but I bet Netflix currently isn't, and TLS allows browsers to enforce verification on behalf of the user. Without verification, you could potentially exploit the browser with malformed audio or video data, or freak the user out by injecting some other video...
I imagine a side-channel attack could reveal some of this information. Fingerprint bandwidth usage over time. Movies consume less bandwidth when scenes move slowly and HTTPS doesn't do a good job of hiding this information.
>> I was just thinking about this, and due to the use of variable-bitrate encoding
As I recall, monitoring total home power consumption could also identify what you're watching on TV. That was on the old CRT sets, not sure about now.
Gotta love side channel attacks.
As I recall, monitoring total home power consumption could also identify what you're watching on TV. That was on the old CRT sets, not sure about now.
Gotta love side channel attacks.
Some HDTVs have dynamic LED backlighting that is probably a power side channel.
This is why I only drink rainwater and grain alcohol, and use pure class A amplifiers, Mandrake.
And you know that can be tracked as well - http://www.academia.edu/5585470/Hydrological_Studies_Using_I...
In the HTML5 context, adaptive streaming is done with the MSE API. This involves fetching the media data using XHR. Since mixed-content XHR is blocked, if the app is on an https origin, the media data has to come from https origins, because cross-scheme CORS is blocked for security reasons.
You want to have the app on https for several reason, starting with the security of session cookies, but in the case of DRM, there's a reason for browsers to seek to limit the EME API (the JS API for communicating with the DRM component) to https (though AFAIK no browser does so at the moment):
DRM schemes usually implement node locking, i.e. anti-cloneability of the DRM state from one device to another. The obvious way to do this involves keys that no other device has. If information about the possession of a persistent key that no other device has is exposed to a network eavesdropper (e.g. by presenting, in the clear, a certificate for a unique key), the eavesdropper can use that piece of information to track devices and, therefore, users.
While this problem could be mitigated in the design of the DRM protocol, to move addressing the problem out of what the DRM component controls to what the browser controls, limiting EME to https is the most straight-forward solution that can be implemented on the browser level. And as noted above, this has the side effect of limiting the MSE-feeding XHR to https, which means the media data needs to be served over https.
You want to have the app on https for several reason, starting with the security of session cookies, but in the case of DRM, there's a reason for browsers to seek to limit the EME API (the JS API for communicating with the DRM component) to https (though AFAIK no browser does so at the moment):
DRM schemes usually implement node locking, i.e. anti-cloneability of the DRM state from one device to another. The obvious way to do this involves keys that no other device has. If information about the possession of a persistent key that no other device has is exposed to a network eavesdropper (e.g. by presenting, in the clear, a certificate for a unique key), the eavesdropper can use that piece of information to track devices and, therefore, users.
While this problem could be mitigated in the design of the DRM protocol, to move addressing the problem out of what the DRM component controls to what the browser controls, limiting EME to https is the most straight-forward solution that can be implemented on the browser level. And as noted above, this has the side effect of limiting the MSE-feeding XHR to https, which means the media data needs to be served over https.
I think the context here is that Netflix does not want to serve video over HTTPS but some browsers are talking about restricting EME DRM so that only HTTPS origins can use it.
HTTP can be used not only for spying on you, but for attacking you with malware as well (or even something as "innocent" as being injected with the ISP's own ads):
https://citizenlab.org/2014/08/cat-video-and-the-death-of-cl...
https://citizenlab.org/2014/08/cat-video-and-the-death-of-cl...
Yes, I agree with you, but in this specific instance, when the DRM key has been transferred securely -- and when everything in the pipeline will fail when the content is messed with -- what is the point of HTTPS?
The "Great Cannon" is probably the best demonstration of the perils of unencrypted transport that we could have hoped for.
You'd be surprised how many ISPs want to MITM media.
It's not about encrypting video files specifically, but:
* Not leaking any tracking or user-identifiable information. EME DRM plug-ins track users, but browsers don't have control over this (and I guess it may even be illegal to audit EME DRM security due to DMCA), so it's best to have it on HTTPS just in case.
* All new browser features require HTTPS (and EME DRM plug-ins are new-ish), because browser vendors want to deprecate HTTP, and don't want headache of mixed content or HTTPS alternatives built on DRM.
* Not leaking any tracking or user-identifiable information. EME DRM plug-ins track users, but browsers don't have control over this (and I guess it may even be illegal to audit EME DRM security due to DMCA), so it's best to have it on HTTPS just in case.
* All new browser features require HTTPS (and EME DRM plug-ins are new-ish), because browser vendors want to deprecate HTTP, and don't want headache of mixed content or HTTPS alternatives built on DRM.
Saying "EME DRM plug-ins track users" is misleading. Rather, an EME DRM plug-in designed only to fulfill its DRM mission in a straight-forward way without specifically trying to also fulfill a privacy mission is likely to have the side effect of exposing identifiers that network eavesdroppers could use for tracking.
[deleted]
EME stands for Encrypted Media Extensions http://www.w3.org/TR/2015/WD-encrypted-media-20150331/
I'm not familiar with AWS, but wouldn't a Cavium based card make this a lot easier for Netflix?
They do not use AWS for streaming, just for the control plane.
I was at the talk and they did not mention hardware offload, but there are a lot of problems with having full control.
I was at the talk and they did not mention hardware offload, but there are a lot of problems with having full control.
Maybe if they're having rouble with EME + HTTPS they should stop being defective by design and abandon their mistake of using EME.
It's a bit hard for me to reconcile "<1% of the CPU load" (Google) with "up to 50% capacity hit" (Netflix).