Comcast is definitely throttling Netflix, and it’s infuriating(mattvukas.com)
mattvukas.com
Comcast is definitely throttling Netflix, and it’s infuriating
http://mattvukas.com/2014/02/10/comcast-definitely-throttling-netflix-infuriating/
14 comments
...when you have to make new peering agreements, build new pipes or create special infrastructure just for one content provider that doesn't pay you a cent...
An ISP can upgrade their equipment for free to receive YouTube traffic via peering or they can upgrade their equipment and pay for transit to receive the same traffic. Given that the traffic is going to be received anyway, peering is by far the cheapest way to accept it.
And if we imagine a world where there were 100 niche video sites instead of YouTube, the total bandwidth would probably be the same but ISPs would be paying for transit to receive it. Google and Netflix's economies of scale benefit ISPs as well.
An ISP can upgrade their equipment for free to receive YouTube traffic via peering or they can upgrade their equipment and pay for transit to receive the same traffic. Given that the traffic is going to be received anyway, peering is by far the cheapest way to accept it.
And if we imagine a world where there were 100 niche video sites instead of YouTube, the total bandwidth would probably be the same but ISPs would be paying for transit to receive it. Google and Netflix's economies of scale benefit ISPs as well.
Last I worked at a ISP, Google actually DOES provide YouTube caching servers to ISPs that want it - they bring you a black box machine which you plug into your backbone switch that does transparent caching for users. Getting one has significantly reduced load on internatinal peering lines (I'm from a small EU country) and made quite a differece for our customers as well (suddenly YouTube started working smoothly for most of them while on other ISPs it sometime still stutter).
I guess same thing can be provided by Netflix as well.
I guess same thing can be provided by Netflix as well.
True. According to RKearney in this thread, Netflix does that too. But isn't that, too, a break of network neutrality? The moment you start accepting content-provider specific equipment, aren't you preferring them over those that don't provide equipment?
Wouldn't accepting equipment from one provider compel you to accept from others too in order to stay neutral.
Wouldn't accepting equipment from one provider compel you to accept from others too in order to stay neutral.
I don't see anything that says they wouldn't accept equipment?
"If your endpoint drives enough traffic, we're open to solutions" doesn't seem silly.
What about ISPs that provide mirrors? An example I've used before was Australian ISPs providing steam, linux, etc mirrors to save their transoceanic costs. Is that still neutral?
"If your endpoint drives enough traffic, we're open to solutions" doesn't seem silly.
What about ISPs that provide mirrors? An example I've used before was Australian ISPs providing steam, linux, etc mirrors to save their transoceanic costs. Is that still neutral?
The way I see it is that once content providers start paying the ISPs, the end users become the product instead of the customer. And when that happens, a lot of perverse incentives tend to come into play and everyone suffers.
The customers (end users) are paying for access to content. Everyone is browsing in YouTube. If the ISP can't afford the connections to provide that, they need to raise their prices.
The customers (end users) are paying for access to content. Everyone is browsing in YouTube. If the ISP can't afford the connections to provide that, they need to raise their prices.
Forgive my ignorance, but why is this such a non-trivial task?
While it seems some management might be required to sufficiently route high traffic partners, does it really require 'building new pipes' or 'special infrastructure' that has a non-trivial cost?
I thought the wonderful aspect of the Internet is such tasks and distribution could be had for very, very cheap.
While it seems some management might be required to sufficiently route high traffic partners, does it really require 'building new pipes' or 'special infrastructure' that has a non-trivial cost?
I thought the wonderful aspect of the Internet is such tasks and distribution could be had for very, very cheap.
Let's say your customers are complaining that youtube is slow. Let's say, you found out that it's because the peering partner of your peering partners peering partner is congested.
To fix this, you can try to talk to that any of the 3 parties involved, or you find a new peering agreement (if possible), or you actually start paying for the traffic.
Or you just leave it as-is, hoping that at some point somebody closer to the congestion starts complaining, which eventually is going to happen.
This certainly is the cheapest route and this is what we honestly can expect considering that all we are promised is 'best effort'. Everything beyond that is really cool to have, but also comes with some expense for the providers.
To fix this, you can try to talk to that any of the 3 parties involved, or you find a new peering agreement (if possible), or you actually start paying for the traffic.
Or you just leave it as-is, hoping that at some point somebody closer to the congestion starts complaining, which eventually is going to happen.
This certainly is the cheapest route and this is what we honestly can expect considering that all we are promised is 'best effort'. Everything beyond that is really cool to have, but also comes with some expense for the providers.
If they treated it as getting paid per megabyte delivered, they would love to make all these interconnects and such.
They fact that they chose to sell 'unlimited' data and look at network investment as a waste is not unfair to them in any way. It's a self-imposed illusion of a problem.
They fact that they chose to sell 'unlimited' data and look at network investment as a waste is not unfair to them in any way. It's a self-imposed illusion of a problem.
But they are NOT paid per megabyte delivered. The moment they start doing that customers complain even more (remember the Comcast 300G limit uproar).
So charging customers per MB is out and charging content providers per MB is out too.
Providing crappy service or investing significant amounts of money and time is all that's left and considering the local monopolies, crappy service is what we get as that's the least amount of trouble for the providers.
So charging customers per MB is out and charging content providers per MB is out too.
Providing crappy service or investing significant amounts of money and time is all that's left and considering the local monopolies, crappy service is what we get as that's the least amount of trouble for the providers.
I am not sure that charging per MB would actually be a problem except they would want to charge outrageous prices instead of fair prices.
> another provider here in Switzerland has a special solution in place for youtube after talking with Google
It's my understanding that this is actually pretty common: https://peering.google.com/about/ggc.html
It's my understanding that this is actually pretty common: https://peering.google.com/about/ggc.html
This kind of test has been discussed at Ars Technica in the last few days. http://arstechnica.com/information-technology/2014/02/verizo... It doesn't conclusively prove anything since the Netflix and VPN traffic take different paths, only one of which may be congested. In particular, I suspect the encryption has nothing to do with it. Ultimately I think this is the same old problem: http://www.internap.com/2010/12/02/peering-disputes-comcast-... but as Netflix increases bitrate and number of subscribers it gets worse.
An alarming number of people don't seem to understand the different between throttling and congestion.
Comcast does not participate in Netflix's OpenConnect project. If Comcast wanted to, they could receive a 4U server from Netflix (multiple ones, in fact, for each densely populated area they service) completely free of cost. Obvious costs for power and rack space apply, although the appliance itself is free.
This would then allow your Netflix viewing to go to a 4U server at a Comcast datacenter or colocation in the area instead of having the entire South East US try to fit through a small pipe at Marietta, GA.
Comcast's stubbornness to participate in the OpenConnect project is the reason for slowness, not throttling. Of course it's going to be faster over a VPN because whatever endpoint you're connecting to most likely takes a different AS path to Netflix.
Comcast does not participate in Netflix's OpenConnect project. If Comcast wanted to, they could receive a 4U server from Netflix (multiple ones, in fact, for each densely populated area they service) completely free of cost. Obvious costs for power and rack space apply, although the appliance itself is free.
This would then allow your Netflix viewing to go to a 4U server at a Comcast datacenter or colocation in the area instead of having the entire South East US try to fit through a small pipe at Marietta, GA.
Comcast's stubbornness to participate in the OpenConnect project is the reason for slowness, not throttling. Of course it's going to be faster over a VPN because whatever endpoint you're connecting to most likely takes a different AS path to Netflix.
If as a provider you start installing equipment from a third party content provider, is that still called "network neutrality" though?
Why would you accept equipment from Netflix but not $STARTUP? Would accepting equipment from Netflix also force you to accept equipment from $OTHERSTARTUP (talking about http://www.avc.com/a_vc/2014/01/vc-pitches-in-a-year-or-two.... here)? What are the security implications?
Why would you accept equipment from Netflix but not $STARTUP? Would accepting equipment from Netflix also force you to accept equipment from $OTHERSTARTUP (talking about http://www.avc.com/a_vc/2014/01/vc-pitches-in-a-year-or-two.... here)? What are the security implications?
Akamai and Google already do this. When you're pushing terrabits per second of data onto an ISP's network they usually agree to do something like this.
It's a win for everyone. ISP has to peer less traffic, customers get a faster experience, and Google/Akamai/Netflix benefit from reduced load.
It's purely speculation on my part, but I believe the only reason we haven't seen OpenConnect adopted at Comcast is because Comcast has everything to lose from making Netflix fast because they're also a cable company. This is where the net neutrality debate comes into play. When Comcast starts preventing the growth of third party services that don't contribute to their bottom line.
It's a win for everyone. ISP has to peer less traffic, customers get a faster experience, and Google/Akamai/Netflix benefit from reduced load.
It's purely speculation on my part, but I believe the only reason we haven't seen OpenConnect adopted at Comcast is because Comcast has everything to lose from making Netflix fast because they're also a cable company. This is where the net neutrality debate comes into play. When Comcast starts preventing the growth of third party services that don't contribute to their bottom line.
What do you mean 'of course it's going to be faster over a VPN'? The VPN is taking a non-optimal route, it should in almost all cases be of similar or slower speed. If the VPN path is fastest then comcast should be sending data in that direction.
Call it congestion if you want but it's intentional congestion caused by intentionally wrong routing.
Call it congestion if you want but it's intentional congestion caused by intentionally wrong routing.
BGP path selection does not route based on link utilization.
What you define as a "non optimal route" isn't what BGP defines as a non optimal route. An ISP can't just send all their customer traffic through any foreign network they choose. The peering location happens to be the "best" path available to Netflix and it won't change until Comcast gets off their high horse and installs Netflix's free caching servers on their network.
What you define as a "non optimal route" isn't what BGP defines as a non optimal route. An ISP can't just send all their customer traffic through any foreign network they choose. The peering location happens to be the "best" path available to Netflix and it won't change until Comcast gets off their high horse and installs Netflix's free caching servers on their network.
An ISP can't send data through any networks at all without some kind of agreement. But that's what makes them an Internet Service Provider, the fact that they have those agreements. When they fail to even properly connect to all Tier 1 networks, that starts to be a misnomer.
I actually get lower latency to Netflix's "delivery hosts" if I bring up a VPN session to IU, simply because my ISP has a private peering with IU (in a building just a few miles from here) and because IU has 100 GbE to a facility in Chicago that Netflix content is served from.
Otherwise, my traffic goes from my ISP to Time Warner to someone else (AT&T? Level3? I don't recall at the moment) before finally hitting Netflix.
Otherwise, my traffic goes from my ISP to Time Warner to someone else (AT&T? Level3? I don't recall at the moment) before finally hitting Netflix.
> Call it congestion if you want but it's intentional congestion caused by intentionally wrong routing.
Something tells me you aren't the operator of a network that has BGP sessions with several other networks.
Something tells me you aren't the operator of a network that has BGP sessions with several other networks.
Why do you assume that? Comcast can't just route traffic through IU's private network
If they can't route almost all traffic in the fastest direction, then something is wrong with their network on a technical level. If IU has a massively superior route, then Comcast is negligent to nether peer nor lay their own line along a similar path.
[deleted]
Google has a similar program for ISPs: Google Global Cache https://peering.google.com/about/ggc.html
I still see a problem with his testing in that he might be connecting to a server in a different area which is less congested when he connects through his school's VPN.
A proper conclusive test would ensure connection to the same Netflix IP somehow over a VPN as his home connection.
A proper conclusive test would ensure connection to the same Netflix IP somehow over a VPN as his home connection.
I live in the same town as the author. Comcast is not my ISP but is the ISP for several of my friends, including friends whom I stayed with for about six weeks recently when I was (for the most part) incapacitated. I can't be 100% certain (I'm going from memory) but I'm almost positive that Netflix traffic did come from the same facility that the author is hitting when connected to the school's VPN. (If it wasn't 2:33 a.m. I'd go verify.)
As I mentioned in another comment, however, the .edu is almost certainly better connected to this facility than Comcast is.
As I mentioned in another comment, however, the .edu is almost certainly better connected to this facility than Comcast is.
Being an ISP is not a natural monopoly. This premise is part of the problem, ISPs are being protected from competition when they should not be.
From the Wiki entry the author quotes:
"A natural monopoly is a monopoly in an industry in which it is most efficient (involving the lowest long-run average cost) for production to be concentrated in a single firm."
Competition in the ISP market drives costs lower as it forces the companies to become more efficient, and increases the value proposition offered to customers (greater speeds at an equal or lesser price, thanks Google Fiber!). In fact, I think it would be easy to prove comprehensively that ISPs are in no way natural monopolies.
If the author was right, and ISPs were a natural monopoly, then there would be nothing to discuss, the only practical solution would be to install government regulated monopolies in every location across the nation to achieve the absolute lowest cost and consumer price. Google demolished this line of thinking very handily, they showed how instantaneously the government protected telecoms respond to competition in a positive way.
The solution is to make it illegal nationally, via Federal law, for any municipality to ever create or encourage an ISP monopoly; and specifically the exact opposite should occur, competition should be heavily encouraged everywhere. The FCC should be mandated with keeping competition lanes wide open in the ISP market, at all times, across the nation. Standards for how cities and towns handle multiple ISPs for infrastructure should be established. Simply: you shall create no law restricting ISP competition or granting monopoly status, period.
From the Wiki entry the author quotes:
"A natural monopoly is a monopoly in an industry in which it is most efficient (involving the lowest long-run average cost) for production to be concentrated in a single firm."
Competition in the ISP market drives costs lower as it forces the companies to become more efficient, and increases the value proposition offered to customers (greater speeds at an equal or lesser price, thanks Google Fiber!). In fact, I think it would be easy to prove comprehensively that ISPs are in no way natural monopolies.
If the author was right, and ISPs were a natural monopoly, then there would be nothing to discuss, the only practical solution would be to install government regulated monopolies in every location across the nation to achieve the absolute lowest cost and consumer price. Google demolished this line of thinking very handily, they showed how instantaneously the government protected telecoms respond to competition in a positive way.
The solution is to make it illegal nationally, via Federal law, for any municipality to ever create or encourage an ISP monopoly; and specifically the exact opposite should occur, competition should be heavily encouraged everywhere. The FCC should be mandated with keeping competition lanes wide open in the ISP market, at all times, across the nation. Standards for how cities and towns handle multiple ISPs for infrastructure should be established. Simply: you shall create no law restricting ISP competition or granting monopoly status, period.
The OP may be using it wrong, but I still think at this point, the last mile network is a natural monopoly. Where I live, city-owned power companies have built single fiber-to-the-building networks that are the open for ISP competition inside the network (I.e. the ISP you pick just delivers the bandwidth/peering). In my view this is the optimal compromise. This wasn't true just a few years ago when DSL and Cable tech was still competitive, but at this point it seems pretty clear that fiber infrastructure isn't going to be beat.
Competition does encourage efficiency, but competing ISPs are starting with lower efficiency because their infrastructure passes so many homes that aren't connected.
Pretty much no one understands how Google Fiber can be profitable at its prices, so unless they open their books or someone replicates that business model I'm not going to call it a solution.
Pretty much no one understands how Google Fiber can be profitable at its prices, so unless they open their books or someone replicates that business model I'm not going to call it a solution.
A list of some things that can go wrong:
* You could have gotten a lousy route due to poor traffic management or a network issue by your ISP, the transit providers, or a CDN
* You could have been given an overloaded server or POP by a CDN's DNS
* Your ISP could have a netflix or third party caching appliance, and it could be overloaded
* Your ISP could be under-backhauled, under-peered, or under-transited in any points along your route
* Your ISP's transit provider(s) could be under-peered, under-backhauled, or under-transited if they are not a tier-1 network. See also poor traffic management and network issues.
* Any party along the way to the CDN and to the origin could be under-peered, under-backhauled, or under-transited. See also poor traffic management and network issues.
Using a VPN proves very little. Various BGP and DNS tricks are used in the bullets above, and a VPN can drastically change the parameters of all of them.
* You could have gotten a lousy route due to poor traffic management or a network issue by your ISP, the transit providers, or a CDN
* You could have been given an overloaded server or POP by a CDN's DNS
* Your ISP could have a netflix or third party caching appliance, and it could be overloaded
* Your ISP could be under-backhauled, under-peered, or under-transited in any points along your route
* Your ISP's transit provider(s) could be under-peered, under-backhauled, or under-transited if they are not a tier-1 network. See also poor traffic management and network issues.
* Any party along the way to the CDN and to the origin could be under-peered, under-backhauled, or under-transited. See also poor traffic management and network issues.
Using a VPN proves very little. Various BGP and DNS tricks are used in the bullets above, and a VPN can drastically change the parameters of all of them.
sure, all of that is possible. but honestly how likely is it that some or all of those things are the case on the direct path from netflix/youtube through the ISP in question to so many people's homes? and yet the problem spot is mysteriously avoided when using a VPN (not any particular VPN, but pretty much any of them, judging by most of the people who have spoken out about this problem recently). the problem so effectively avoided by changing the route from the optimal(ish) path to one that is almost guaranteed to be worse from a network flow standpoint.
sorry, I don't buy any apologizing for comcast/verizon failing to deliver HD video from netflix/youtube yet somehow able to do it under the same network conditions (same time) just this time through an encrypted VPN. if it was an isolated report... maybe. this is far too widespread to be anything but intentional.
sorry, I don't buy any apologizing for comcast/verizon failing to deliver HD video from netflix/youtube yet somehow able to do it under the same network conditions (same time) just this time through an encrypted VPN. if it was an isolated report... maybe. this is far too widespread to be anything but intentional.
Well, under-provisioned backhaul, peering, and/or transit would cause exactly this. And the blame trail could be quite complex. I'm not apologizing, but commentary by almost anyone other than the network architects and peering coordinators of these organizations are pretty baseless speculations.
Disclaimer: I work for a major CDN. Netflix is trying to ween off of us and our competitors, which has contributed to lesser experience. Their model of co-locating gear for free doesn't make much sense to ISPs in the grand scheme of things.
Disclaimer: I work for a major CDN. Netflix is trying to ween off of us and our competitors, which has contributed to lesser experience. Their model of co-locating gear for free doesn't make much sense to ISPs in the grand scheme of things.
So, let me tell you a story about Comcast and network engineering. Actually, no, I'll skip that.
Basically - Comcast runs all of its ports hot. I would say its safe to say what you're seeing is Comcast sucking at paying for upgrades rather than anything specifically malicious.
Basically - Comcast runs all of its ports hot. I would say its safe to say what you're seeing is Comcast sucking at paying for upgrades rather than anything specifically malicious.
FWIW, the author's school has tens of gigabits per seconds (and it may even be 100 Gbps by now) to a facility in which Netflix has a presence. Comcast, AIUI, doesn't.
It's really simple, people. The links that Comcast is carrying Netflix traffic over is getting saturated during peak hours. I am certainly not a fan of Comcast but I don't think they are actively doing anything to screw with Netflix traffic, they just aren't doing anything to fix it.
This could be tested easily by pinging multiple destinations (besides Netflix) on peak hours
Only if you can ensure that those pings are traversing the same circuits that the Netflix traffic is.
(And, even if you can, you can't be sure of the reverse path -- something so many people don't understand when doing traceroutes.)
(And, even if you can, you can't be sure of the reverse path -- something so many people don't understand when doing traceroutes.)
[deleted]
Comcast is still supposed to be bound by net neutrality rules, unlike other ISPs, because they agreed to follow them until 2018 as part of getting their merger with NBC approved.
While comcast sucks for its price and monopoly, i think netflix is at fault here. I did a similar test a while back using my company cisco vpn which i personally administer :) and found netflix sucked big time with or without vpn around 9pm any day. Could never get even DVD quality on my 1080p despite being in San Jose, CA with 25Mbps+ crapcast
I wouldn't be so sure, I AnyConnect sucks and Comcast has been heavily congested in SJC for months. I have some graphs I should show you...
The author points out that netflix cannot see what is going on over the tunnel: Actually, if Comcast wanted to they could tell what you are doing by using deep packet inspection. Certain firewall vendors have found ways to figure out what you are doing over encrypted tunnels based on protocol pattern/finger printing.
They could probably guess he was watching streaming video. It'd be tough for deep packet inspection to tell that it's netflix though.
how could they guess he's streaming a video and not downloading a large file?
Timing of packets going both ways. Netflix is a bad example since they use HTTP to wing packets, but with a UDP connection you won't see the regular ACKs heading back upstream.
In this case, the VPN being used is a Juniper SSL VPN, which runs over 443/TCP. Comcast would see TCP ACKs being sent back to the VPN endpoint but they obviously can't see the encrypted data.
One other note: the author's school (the entity he brought up the VPN session to) manages many of the largest R&D networks in the country. As such, it is extremely well-connected to, well, just about everything (including Netflix's delivery networks).
Over time, this was fixed, only to return later, at which point it was fixed again.
Huge kudos to my provider (Cablecom in Switzerland) for actually fixing this and even in a somewhat timely manner.
But.
As more and more of the internet users traffic gets centralized to a few big services (Netflix, YouTube, to some extent Facebook), the usual unbureaucratic peering agreements start to become less and less feasible as traffic starts to be centralized with a few very major content providers.
Not only that: some players might start to produce so much traffic that it starts to become useful to go as far as to start doing special proxy solutions for specific content providers (another provider here in Switzerland has a special solution in place for youtube after talking with Google).
At which point does all that effort in order to have your end-users be able to access some of these huge content providers stop being worth it in the name of net neutrality?
Providing network neutrality is easy when everything just works, but when you have to make new peering agreements, build new pipes or create special infrastructure just for one content provider that doesn't pay you a cent, then that whole neutrality thing becomes very inconvenient and, I have to say, unfair to the providers.
Yes. I really, really want network neutrality, but I see that aside of all the political and business-model-preservation issues, we are running into technical issues too and those need solving too.