Observations of an Internet Middleman(blog.level3.com)
blog.level3.com
Observations of an Internet Middleman
http://blog.level3.com/global-connectivity/observations-internet-middleman/
172 comments
Disclosure: I work at a small ISP. We mostly compete with Frontier DSL and Comcast Business Class.
If you're buying residential it's safe to assume you're purchasing a burst-able speed and downloads/uploads aren't going to sustain that for hours on end.
Much in the same way that a Utility Company probably isn't gathering enough water for everyone to be maxing their pipes 24/7, residential ISP's don't operate under the assumption that every consumer will have maxed their connection at the same time, and they shouldn't have too. Smart traffic shaping and peering arrangements gives ISP's room to compete.
If consumers could afford a dedicated 35x15 connection they'd have a T1. When you're buying copper it's safe to assume it's going to be over saturated.
The real problem is ISP's for the most part don't compete. They zone of sections for eachother and rack in as much dough as they can. Consumers aren't informed about the quality of different types of connections vs others, and buy based off the cheapest Mbps down/up they can get.
If you're buying residential it's safe to assume you're purchasing a burst-able speed and downloads/uploads aren't going to sustain that for hours on end.
Much in the same way that a Utility Company probably isn't gathering enough water for everyone to be maxing their pipes 24/7, residential ISP's don't operate under the assumption that every consumer will have maxed their connection at the same time, and they shouldn't have too. Smart traffic shaping and peering arrangements gives ISP's room to compete.
If consumers could afford a dedicated 35x15 connection they'd have a T1. When you're buying copper it's safe to assume it's going to be over saturated.
The real problem is ISP's for the most part don't compete. They zone of sections for eachother and rack in as much dough as they can. Consumers aren't informed about the quality of different types of connections vs others, and buy based off the cheapest Mbps down/up they can get.
in the same way that a Utility Company probably isn't gathering enough water for everyone to be maxing their pipes 24/7, residential ISP's don't operate under the assumption that every consumer will have maxed their connection at the same time
That's not what's happening, though. What's happening is that the ISPs have not provisioned enough capacity to serve the actual aggregate demand at peak times. The hydraulic analogy would be if water pressure dropped every day between 0800 and 0900 because everyone was having their morning shower. That would not be acceptable to customers, and nor is the situation with ISPs.
That's not what's happening, though. What's happening is that the ISPs have not provisioned enough capacity to serve the actual aggregate demand at peak times. The hydraulic analogy would be if water pressure dropped every day between 0800 and 0900 because everyone was having their morning shower. That would not be acceptable to customers, and nor is the situation with ISPs.
Exactly. Great analogy!
What's more, the utility company is getting water through a pipe that goes to a middleman near a river, but that pipe doesn't have enough capacity to deliver the water necessary for all utility customers to take their morning shower at the same time. So what does the utility company do? It blames the middleman and the river for "sending too much water"... and demands payment from them!!!
What's more, the utility company is getting water through a pipe that goes to a middleman near a river, but that pipe doesn't have enough capacity to deliver the water necessary for all utility customers to take their morning shower at the same time. So what does the utility company do? It blames the middleman and the river for "sending too much water"... and demands payment from them!!!
As I see the graph in the post it looks like the customers have full pressure only between 3 and 4AM. In the rest part of the day they are struggling to get their shower.
The congestion in the news is at the interconnection between networks, not the last mile. This is why Netflix's deal with Comcast resulted in immediate relief without a massive rollout to upgrade last mile infrastructure.
Cable providers have a large amount of capacity provisioned to their customers and are paid a fortune by their subscribers and municipalities, using the same network for data and cable services. They just want you to use their services as opposed to a competitor. Not only that they have turned the tables forcing content providers to pay them to deliver bits.
With respect the economics of a small ISP reselling transit aren't comparable to this situation.
Cable providers have a large amount of capacity provisioned to their customers and are paid a fortune by their subscribers and municipalities, using the same network for data and cable services. They just want you to use their services as opposed to a competitor. Not only that they have turned the tables forcing content providers to pay them to deliver bits.
With respect the economics of a small ISP reselling transit aren't comparable to this situation.
[deleted]
What is cheaper? Actually giving customers the bandwidth they are paying for, or maintaining a political lobby apparatus that allows you to remain a monopoly in your industry? Do the math.
The math has already been done. Lobbying has a 22,000% ROI. No brainer.
http://www.npr.org/blogs/money/2012/01/06/144737864/forget-s...
http://www.npr.org/blogs/money/2012/01/06/144737864/forget-s...
For once, I am glad to be living in a densely populated European country. We can easily switch ISP's if one decides to pull these kinds of dirty tricks.
Well, except that the local telco (e.g. BT in the UK) almost certainly has monopoly over the last mile of your connection, regardless of who your ISP is (they rent the last mile from e.g. BT Openreach or maybe TalkTalk) and it's not uncommon for them to refuse to fix congestions at their exchanges (or even admit their presence) for extended periods. So, you know, it's not like it's all utopistic here or something.
The worst bit is that OpenReach are publishing data where they will wire the cabinets, but are hiding information when the cabinets will be wired to your house.
They are actually going to court for this false advertising.
They are actually going to court for this false advertising.
Don't Virgin use their own last mile for fibre?
If by their own you mean one of numerous other cable companies that they purchased then yes. There used to be a plethora of small cable companies in the UK. Now there is only Virgin.
The widespread availability of high quality ADSL does seem to do a good job of keeping them honest though. I've been a pretty satisfied customer of them for many years now.
The widespread availability of high quality ADSL does seem to do a good job of keeping them honest though. I've been a pretty satisfied customer of them for many years now.
Virgin offer a "cable" service which is available where they have cable television (not a wide distribution area), and that this is what they refer to as their fibre service.
Virgin also resell the same ADSL that everyone else does (complete with BT line rental).
Virgin also resell the same ADSL that everyone else does (complete with BT line rental).
While it's true that the average population density of the US is low compared to much of the rest of the world, that is due to including the US's vast empty spaces, mainly in the West. The vast majority of US citizens live in densely populated areas - just like the rest of the First World, only with crappier internet.
The population density is still radically different even when you cut the vast empty spaces between cities in the west.
I live in a midwestern suburb. In the land area that my house uses, you'd have 10-20 families in Madrid. Our idea of dense shopping is a two story mall, not stores under every residential area. And don't get me started with mandated parking spots in strip malls that aren't ever half full.
So unless you live in Manhattan or downtown Chicago, you don't see the same levels of population density.
Single fami
I live in a midwestern suburb. In the land area that my house uses, you'd have 10-20 families in Madrid. Our idea of dense shopping is a two story mall, not stores under every residential area. And don't get me started with mandated parking spots in strip malls that aren't ever half full.
So unless you live in Manhattan or downtown Chicago, you don't see the same levels of population density.
Single fami
Population density is still a red-herring in this discussion though. Internet connections in Manhattan are worse than those in small cities in Romania, yet Manhattan is 3-12 times more dense than Timișoara and Bucharest.
We have choices in most large markets in the US but the one common factor is horrible customer service. You're likely to pay a lot for high performance.
yeah, I too thought the EU was the land of (regulation) milk and honey yet in Germany you have Deutsche Telekom and those that resell Deutsche Telekom with non-performance related value adds.
funny how the blau.de "3.5G" connection is wildly faster and less latent than the cable modem connection. actually it's kinda sad: 5GB for €10/month?
funny how the blau.de "3.5G" connection is wildly faster and less latent than the cable modem connection. actually it's kinda sad: 5GB for €10/month?
I enjoy seeing those MRTG/RRDtool graphs everywhere. I find it so surreal that $X Billions in infrastructure, that is forwarding $Y Billion of internet traffic. All monitored by a single tool created by one guy from Switzerland.
Since this is HN: yes I realize that there has been substantial work subsequently, and that cacti/munin/nagios etc.. are more commonly used.
Since this is HN: yes I realize that there has been substantial work subsequently, and that cacti/munin/nagios etc.. are more commonly used.
I am surprised that the hobby project by the funny sounding guy from Finland did not mentally prepare you for this.
It's crazy to think that Rancid, MRTG, & RRD are all fundamentally underpinning of all of these major carriers.
What's crazy about it?
quote from the author, "rancid is named rancid because it stinks." it covers 80% of what you want it to do, but is indispensable because it is so lightweight and functional.
more developed tools exist but networks are not interested (by in large) in integrating/developing new ones because they fool themselves into thinking these are good enough.
more developed tools exist but networks are not interested (by in large) in integrating/developing new ones because they fool themselves into thinking these are good enough.
Here's the part that doesn't make sense to me, and hopefully someone can explain it:
Netflix pays Level 3 and Cogent to connect them to Comcast's network. Comcast claims that only the Level 3 connection is saturated, and that Netflix is sending all their bandwidth over Level 3 because it's cheaper for them.
Doesn't Level 3 buy a contract from Comcast that says "we get to send this much data per month"? If Netflix (or Level 3) tries to push more data through that, which causes congestion, it seems like a contractual fact that Comcast is either holding up their end of the bargain or not.
If that's true, and assuming that Comcast is transferring the contractually agreed upon part, isn't this actually Netflix and Level 3's fault? Isn't it reasonable to assume that Netflix would need to either use another entry point into Comcast's network, or build one?
If Netflix/Level 3 congest one entry point, so movies stream slowly for Comcast users, it doesn't seem like Comcast is being unneutral, in the sense of packet inspection and routing based on content/source.
The problem seems upstream from the last mile networks, and it's very unclear to me whose fault it is, and if this even has anything to do with net neutrality at all (it doesn't seem to).
Please correct me where I'm wrong, I'm sure I am!
Netflix pays Level 3 and Cogent to connect them to Comcast's network. Comcast claims that only the Level 3 connection is saturated, and that Netflix is sending all their bandwidth over Level 3 because it's cheaper for them.
Doesn't Level 3 buy a contract from Comcast that says "we get to send this much data per month"? If Netflix (or Level 3) tries to push more data through that, which causes congestion, it seems like a contractual fact that Comcast is either holding up their end of the bargain or not.
If that's true, and assuming that Comcast is transferring the contractually agreed upon part, isn't this actually Netflix and Level 3's fault? Isn't it reasonable to assume that Netflix would need to either use another entry point into Comcast's network, or build one?
If Netflix/Level 3 congest one entry point, so movies stream slowly for Comcast users, it doesn't seem like Comcast is being unneutral, in the sense of packet inspection and routing based on content/source.
The problem seems upstream from the last mile networks, and it's very unclear to me whose fault it is, and if this even has anything to do with net neutrality at all (it doesn't seem to).
Please correct me where I'm wrong, I'm sure I am!
I think a big problem with your reasoning centers on word choice; you phrased it as Level 3 having a contract that says "we get to send this much data per month", and then Level 3 attempting to send more data than that agreement.
However, you must remember that every single packet that Level 3 sends to the Comcast network is a packet that was requested BY A COMCAST CUSTOMER. Level 3 isn't just deciding to send a bunch of traffic over Comcasts network, they are sending the data requested by Comcast customers to that network.
This is an important distinction, and is usually part of all of these 'settlement-free' peering arrangements. In agreeing to peer with Level 3, I am sure Comcast has an agreement that they will not send any traffic to Comcast's network that is not actually destined for a Comcast customer. Now, if Level 3 were sending traffic to Comcast and expecting Comcast to route, for free, that traffic to another 3rd part network, that would NOT be kosher, and Comcast could fairly ask for either money or for Level 3 to stop sending that traffic to them.
Level 3 is ONLY sending the traffic to Comcast that Comcast is requesting; it is THEIR customers who are creating this demand for Level 3's customer's content. Comcast told their customers that for $X per month, they would get Y amount of internet bandwidth, which will in almost all cases be traffic originating from a non-Comcast network; that is just the nature of the internet. Even though Level 3 is willing to send the data that Comcast's customers want to the Comcast network for free (settlement free), Comcast is instead refusing to accept all the traffic that their own customers are requesting and demanding additional payment to provide the service they already sold and are paid for.
However, you must remember that every single packet that Level 3 sends to the Comcast network is a packet that was requested BY A COMCAST CUSTOMER. Level 3 isn't just deciding to send a bunch of traffic over Comcasts network, they are sending the data requested by Comcast customers to that network.
This is an important distinction, and is usually part of all of these 'settlement-free' peering arrangements. In agreeing to peer with Level 3, I am sure Comcast has an agreement that they will not send any traffic to Comcast's network that is not actually destined for a Comcast customer. Now, if Level 3 were sending traffic to Comcast and expecting Comcast to route, for free, that traffic to another 3rd part network, that would NOT be kosher, and Comcast could fairly ask for either money or for Level 3 to stop sending that traffic to them.
Level 3 is ONLY sending the traffic to Comcast that Comcast is requesting; it is THEIR customers who are creating this demand for Level 3's customer's content. Comcast told their customers that for $X per month, they would get Y amount of internet bandwidth, which will in almost all cases be traffic originating from a non-Comcast network; that is just the nature of the internet. Even though Level 3 is willing to send the data that Comcast's customers want to the Comcast network for free (settlement free), Comcast is instead refusing to accept all the traffic that their own customers are requesting and demanding additional payment to provide the service they already sold and are paid for.
"Level 3 is ONLY sending the traffic to Comcast that Comcast is requesting; it is THEIR customers who are creating this demand for Level 3's customer's content."
This still misses the mark.
Netflix creates a service on Level3 that Comcast customers want. So Netflix/Level3 are responsible for making a product that people want to use, and Comcast customers are responsible for wanting to use that product and increasing bandwidth use. Neither Level3 nor Comcast want to pay more money just because Netflix and Comcast customers want to use more tubes, so both Level3 and Comcast wash their hands of any responsibility for the networks they both run.
In a very real sense, both Level3 and Comcast are equally responsible for the increase in traffic [in this case]. Yet neither seem to accept their share in the cost. If it's because it's hard for Level3 to prove, they should put up some money to shore up the connection, and then write a public letter to Comcast customers telling them how Comcast is shorting them and to get angry.
(Sadly, many customers have no alternative provider, so their only option would be to quit the internet, and nobody's going to do that)
This still misses the mark.
Netflix creates a service on Level3 that Comcast customers want. So Netflix/Level3 are responsible for making a product that people want to use, and Comcast customers are responsible for wanting to use that product and increasing bandwidth use. Neither Level3 nor Comcast want to pay more money just because Netflix and Comcast customers want to use more tubes, so both Level3 and Comcast wash their hands of any responsibility for the networks they both run.
In a very real sense, both Level3 and Comcast are equally responsible for the increase in traffic [in this case]. Yet neither seem to accept their share in the cost. If it's because it's hard for Level3 to prove, they should put up some money to shore up the connection, and then write a public letter to Comcast customers telling them how Comcast is shorting them and to get angry.
(Sadly, many customers have no alternative provider, so their only option would be to quit the internet, and nobody's going to do that)
Yet neither seem to accept their share in the cost.
It seems like Level3 is ready and willing to go. Comcast is just leveraging their monopoly to get out of standard peering practices. They know that by causing a fuss the lookie loos will show up and say "oh my, a controversy, lets split the difference!" Bonus points if you can work in the phrases "right balance" and "fair share."
If it's because it's hard for Level3 to prove, they should put up some money to shore up the connection, and then write a public letter to Comcast customers telling them how Comcast is shorting them and to get angry.
An open letter after giving into their demands? Did you just tell me to go fuck myself? This is that open letter. If they do it after giving into Comcast the lookie loos will be all the more ready and willing to pull "compromises" out of their ass that don't solve anything but making them happy with themselves.
It seems like Level3 is ready and willing to go. Comcast is just leveraging their monopoly to get out of standard peering practices. They know that by causing a fuss the lookie loos will show up and say "oh my, a controversy, lets split the difference!" Bonus points if you can work in the phrases "right balance" and "fair share."
If it's because it's hard for Level3 to prove, they should put up some money to shore up the connection, and then write a public letter to Comcast customers telling them how Comcast is shorting them and to get angry.
An open letter after giving into their demands? Did you just tell me to go fuck myself? This is that open letter. If they do it after giving into Comcast the lookie loos will be all the more ready and willing to pull "compromises" out of their ass that don't solve anything but making them happy with themselves.
> It seems like Level3 is ready and willing to go.
To go where? Not to their checkbook.
First they mention in the article how "usually" networks agree to upgrade parts of their network to handle additional interconnects, and don't pay service fees to each other.
Then they mention saturated ports, not giving any indication as to whether they have even attempted to expand capacity on their side.
We are also left to believe that this nonspecific, non-detailed post is not only completely genuine, but that it leaves out no details. Giving Level3 the benefit of the doubt, and not going based on my own enterprise peering experiences where Level3 consistently showed the worst reliability and performance, it's clear they're only showing you a small slice of the work involved to agree on peering and put in place the upgrades necessary, on both sides.
> An open letter after giving into their demands?
An open letter after actually spending their share of what they need to invest in the upgrade. Which, it appears, they have not.
> Did you just tell me to go fuck myself?
No... Where did you read that?
> If they do it after giving into Comcast the lookie loos will be all the more ready and willing to pull "compromises" out of their ass that don't solve anything but making them happy with themselves.
....You really don't know the meaning of the word compromise? One of them has to meet the other half way. Ideally both of them would do this at the same time, but in the bizarre reality of this business, both of them are being children, neither wanting to move first, it would seem from this writing. Too bad this is just one tiny public PR stunt and probably not the whole story.
Also, what it would solve is the issue at hand, and it would make everyone - both providers, consumers and customers - happy. If you can't see that, perhaps you have the same narrow-minded political pessimism that's causing this mess to begin with.
This whole letter is thinly-veiled fodder for consumers to bitch at their ISPs with. Even the comparisons, like the ones at the end about industry satisfaction, are just meant to feed into the stereotype of the evil broadband provider. The comparison is ridiculous, because broadband providers are held accountable for every network and server fuckup on the entire internet, as well as the general faults of an aging physical network that spans a giant continent.
To go where? Not to their checkbook.
First they mention in the article how "usually" networks agree to upgrade parts of their network to handle additional interconnects, and don't pay service fees to each other.
Then they mention saturated ports, not giving any indication as to whether they have even attempted to expand capacity on their side.
We are also left to believe that this nonspecific, non-detailed post is not only completely genuine, but that it leaves out no details. Giving Level3 the benefit of the doubt, and not going based on my own enterprise peering experiences where Level3 consistently showed the worst reliability and performance, it's clear they're only showing you a small slice of the work involved to agree on peering and put in place the upgrades necessary, on both sides.
> An open letter after giving into their demands?
An open letter after actually spending their share of what they need to invest in the upgrade. Which, it appears, they have not.
> Did you just tell me to go fuck myself?
No... Where did you read that?
> If they do it after giving into Comcast the lookie loos will be all the more ready and willing to pull "compromises" out of their ass that don't solve anything but making them happy with themselves.
....You really don't know the meaning of the word compromise? One of them has to meet the other half way. Ideally both of them would do this at the same time, but in the bizarre reality of this business, both of them are being children, neither wanting to move first, it would seem from this writing. Too bad this is just one tiny public PR stunt and probably not the whole story.
Also, what it would solve is the issue at hand, and it would make everyone - both providers, consumers and customers - happy. If you can't see that, perhaps you have the same narrow-minded political pessimism that's causing this mess to begin with.
This whole letter is thinly-veiled fodder for consumers to bitch at their ISPs with. Even the comparisons, like the ones at the end about industry satisfaction, are just meant to feed into the stereotype of the evil broadband provider. The comparison is ridiculous, because broadband providers are held accountable for every network and server fuckup on the entire internet, as well as the general faults of an aging physical network that spans a giant continent.
> not giving any indication as to whether they have even attempted to expand capacity on their side.
I guess you missed this part then:
"Six of those 12 have a single congested port, and we are both (Level 3 and our peer) in the process of making upgrades – this is business as usual and happens occasionally as traffic swings around the Internet as customers change providers."
I guess you missed this part then:
"Six of those 12 have a single congested port, and we are both (Level 3 and our peer) in the process of making upgrades – this is business as usual and happens occasionally as traffic swings around the Internet as customers change providers."
Didn't miss it. And it would seem that has nothing to do with Comcast. Just read the paragraph again. They have 6 peers who are using a single 10 Gigabit Ethernet port and somehow (who'da thunk?) it is congested. So they are upgrading their one port that is degrading traffic for six peers.
They're right, it is business as usual, and you do have to change ports as traffic flows change. This has nothing to do with Level3 and Comcast's (apparent?) failure to reach consensus on a peering agreement, or if either of them has invested anything in it.
They're right, it is business as usual, and you do have to change ports as traffic flows change. This has nothing to do with Level3 and Comcast's (apparent?) failure to reach consensus on a peering agreement, or if either of them has invested anything in it.
As I understand it there are 6 peers with the following situation:
- Between 1 to 20 ports between Level3 and that peer
- 1 of those ports is congested
- They are handling the upgrade together, which is business as usual.
- Between 1 to 20 ports between Level3 and that peer
- Almost all of those ports are congested
- The peer refuses to augment capacity
> They have 6 peers who are using a single 10 Gigabit Ethernet port
No, they have 6 peers who have a single congested port each. It's a different port for each peer (the article says nothing about multiple peers sharing a single port), and I can't tell from the article if the single port for each peer is the only port they have with those peers. It might be, or it might not. Whether it is or not is irrelevant to the key point, which is that these peers are cooperating with Level 3 to upgrade capacity, which is not true of the 6 peers who have had permanent congestion for over a year.
No, they have 6 peers who have a single congested port each. It's a different port for each peer (the article says nothing about multiple peers sharing a single port), and I can't tell from the article if the single port for each peer is the only port they have with those peers. It might be, or it might not. Whether it is or not is irrelevant to the key point, which is that these peers are cooperating with Level 3 to upgrade capacity, which is not true of the 6 peers who have had permanent congestion for over a year.
> Then they mention saturated ports, not giving any indication as to whether they have even attempted to expand capacity on their side.
How can they upgrade the capacity on their side? Suppose they drive 100GBit to the router that connects to Comcast. They can still only push 10GBit to Comcast if Comcast only make a single 10GBit port available. What they tell us is that they are ready and willing to upgrade their side and Comcast are not ready/willing to upgrade theirs.
> ....You really don't know the meaning of the word compromise? One of them has to meet the other half way. Ideally both of them would do this at the same time, but in the bizarre reality of this business, both of them are being children, neither wanting to move first, it would seem from this writing. Too bad this is just one tiny public PR stunt and probably not the whole story.
"Compromise" is a very bad long term strategy with extortionists, blackmailers, kidnappers and jackasses in general. "Compromising" with someone being an jackass is likely to lead to more jackass behaviour in the future.
How can they upgrade the capacity on their side? Suppose they drive 100GBit to the router that connects to Comcast. They can still only push 10GBit to Comcast if Comcast only make a single 10GBit port available. What they tell us is that they are ready and willing to upgrade their side and Comcast are not ready/willing to upgrade theirs.
> ....You really don't know the meaning of the word compromise? One of them has to meet the other half way. Ideally both of them would do this at the same time, but in the bizarre reality of this business, both of them are being children, neither wanting to move first, it would seem from this writing. Too bad this is just one tiny public PR stunt and probably not the whole story.
"Compromise" is a very bad long term strategy with extortionists, blackmailers, kidnappers and jackasses in general. "Compromising" with someone being an jackass is likely to lead to more jackass behaviour in the future.
I'm pretty sure the article just stated that Level3 is willing to upgrade their equipment, but the broadband provider(likely comcast) is not. Are you suggesting it is a lie?
I think your issue is here:
"Doesn't Level 3 buy a contract from Comcast that says "we get to send this much data per month"? If Netflix (or Level 3) tries to push more data through that, which causes congestion, it seems like a contractual fact that Comcast is either holding up their end of the bargain or not."
Level 3 doesn't buy the connection from Comcast. Comcast buys a connection from Level 3. Comcast needs to do this because they're selling a connection to the "Internet" to their customers, not a connection to the servers Comcast owns, and to other Comcast customers.
Comcast's paying customers are requesting data from the Internet, data that Comcast doesn't have on its network. To get there those bits travel over Level 3's (or Cogent, or another major provider) network. The issue at hand is not one of Level 3 (or Netflix) trying to PUSH data onto Comcast's network. It is one of Comcast's customers trying to PULL data from Level 3, over a connection that Comcast refuses to upgrade.
Level 3 doesn't buy the connection from Comcast. Comcast buys a connection from Level 3. Comcast needs to do this because they're selling a connection to the "Internet" to their customers, not a connection to the servers Comcast owns, and to other Comcast customers.
Comcast's paying customers are requesting data from the Internet, data that Comcast doesn't have on its network. To get there those bits travel over Level 3's (or Cogent, or another major provider) network. The issue at hand is not one of Level 3 (or Netflix) trying to PUSH data onto Comcast's network. It is one of Comcast's customers trying to PULL data from Level 3, over a connection that Comcast refuses to upgrade.
Part of the issue is the monopoly most broadband providers have in the US and Canada.
If you had two truly distinct cable options, say Comcast and TascCom. Your friends with Comcast say Netflix sux, low quality, lots of problems. Your friends with TascCom say netflix is awesome, great HD content, too bad there's only like 8 3D movies! You'll sign up for TascCom. Comcast's board will get angry they're losing customers, and fix their interconnects. But it's all a dream, there is no TascCom, there's only Comcast.
If you had two truly distinct cable options, say Comcast and TascCom. Your friends with Comcast say Netflix sux, low quality, lots of problems. Your friends with TascCom say netflix is awesome, great HD content, too bad there's only like 8 3D movies! You'll sign up for TascCom. Comcast's board will get angry they're losing customers, and fix their interconnects. But it's all a dream, there is no TascCom, there's only Comcast.
Yeah, this does seem to be a core issue. How can we make it easier for people to start last-mile ISPs? It seems like a pretty capital intensive business to get involved in :)
The core issue isn't at the last mile, but in connecting a last mile network to the internet?
Maybe we make Cable companies share their last mile networks like we did with DSL?
Maybe we make Cable companies share their last mile networks like we did with DSL?
> The core issue isn't at the last mile, but in connecting a last mile network to the internet?
Not really. The cost of the interconnect between e.g. Comcast and Level 3 is immaterial in the cost of operating a network. The only reason there is so much contention there is that it's a choke point where Comcast can try to put up a toll booth.
Not really. The cost of the interconnect between e.g. Comcast and Level 3 is immaterial in the cost of operating a network. The only reason there is so much contention there is that it's a choke point where Comcast can try to put up a toll booth.
> Maybe we make Cable companies share their last mile networks like we did with DSL?
Which killed investment in DSL. Who wants to spend billions on infrastructure they are forced to lease out at wholesale rates to competitors?
Which killed investment in DSL. Who wants to spend billions on infrastructure they are forced to lease out at wholesale rates to competitors?
Yet somehow we manage to spend trillions of dollars on streets, sewers and other infrastructure
Those things are built with public money, not private money, which introduces problems of its own. Once you start using public money to build infrastructure, politics determines the level of spending rather than actual demand.
The American Society of Civil Engineers estimates that we have $3.6 trillion in delayed maintenance and underinvestment of our core infrastructure (water, sewers, bridges, power lines, etc): https://www.wsws.org/en/articles/2013/04/08/infr-a08.html.
I'm not one of those people that believes we shouldn't have public infrastructure, but I do think you have to be cognizant of the trade-offs involved. Take something like Amtrak. Amtrak has an almost $9 billion maintenance backlog on the Northeast Corridor: http://usa.streetsblog.org/2011/06/15/house-plan-to-privatiz.... The NEC is the only part of the whole system that generates an operating profit. A private company would shut down the rest of the network, and try to make the NEC service as attractive as possible for customers. But in a regime where politics decides where the money goes, the operating surplus generated by the NEC instead goes to funding money-losing lines in the rest of the country.
Road building, too, is the result of distorted incentives. As an urbanite who doesn't like to drive, I would spend $0 on highways designed to get suburban commuters into the cities, and spend that money on public transit instead. Surbanites, of course, feel differently. Who decides how that money is allocated? Not the market, but the political system, which at the national level systematically over represents rural and suburban votes.
The situation with telecom companies isn't ideal, but I don't think the dynamics of the telecom market are amenable to the kind of broad political consensus necessary for a successful municipal service. Take water, for example. Everybody needs roughly the same amount of water, and is satisfied with a relatively similar level of water quality. Meanwhile, I'd bet 95% of people would be perfectly happy with 5 mbps service, while a small minority wants gigabit. Do you think the political system is set up to make that small minority happy? If there is anything to learn from how municipalities handle public infrastructure is that when you put it to a vote, the voters will spend as little as possible to get the minimum acceptable level of service. That's exactly what happened to our power and water infrastructure.
The American Society of Civil Engineers estimates that we have $3.6 trillion in delayed maintenance and underinvestment of our core infrastructure (water, sewers, bridges, power lines, etc): https://www.wsws.org/en/articles/2013/04/08/infr-a08.html.
I'm not one of those people that believes we shouldn't have public infrastructure, but I do think you have to be cognizant of the trade-offs involved. Take something like Amtrak. Amtrak has an almost $9 billion maintenance backlog on the Northeast Corridor: http://usa.streetsblog.org/2011/06/15/house-plan-to-privatiz.... The NEC is the only part of the whole system that generates an operating profit. A private company would shut down the rest of the network, and try to make the NEC service as attractive as possible for customers. But in a regime where politics decides where the money goes, the operating surplus generated by the NEC instead goes to funding money-losing lines in the rest of the country.
Road building, too, is the result of distorted incentives. As an urbanite who doesn't like to drive, I would spend $0 on highways designed to get suburban commuters into the cities, and spend that money on public transit instead. Surbanites, of course, feel differently. Who decides how that money is allocated? Not the market, but the political system, which at the national level systematically over represents rural and suburban votes.
The situation with telecom companies isn't ideal, but I don't think the dynamics of the telecom market are amenable to the kind of broad political consensus necessary for a successful municipal service. Take water, for example. Everybody needs roughly the same amount of water, and is satisfied with a relatively similar level of water quality. Meanwhile, I'd bet 95% of people would be perfectly happy with 5 mbps service, while a small minority wants gigabit. Do you think the political system is set up to make that small minority happy? If there is anything to learn from how municipalities handle public infrastructure is that when you put it to a vote, the voters will spend as little as possible to get the minimum acceptable level of service. That's exactly what happened to our power and water infrastructure.
Meanwhile, I'd bet 95% of people would be perfectly happy with 5 mbps service, while a small minority wants gigabit.
Historic and present demand are terrible ways of predicting future demand when it comes to technology. Most people don't know what they want because they're living in yesterday, but wait five years when they see what their early adopter neighbors are doing, then suddenly everybody would be happy with gigabit service, and who needs to upgrade to 10GbE anyway?
You could easily have said that nobody would want more than 128kb/s ISDN, because nobody does anything more intensive than download music, check e-mail, and watch flash animations. Faster speeds made newer services, services that are used by very ordinary people (like Netflix, Hangouts, Skype) possible.
Historic and present demand are terrible ways of predicting future demand when it comes to technology. Most people don't know what they want because they're living in yesterday, but wait five years when they see what their early adopter neighbors are doing, then suddenly everybody would be happy with gigabit service, and who needs to upgrade to 10GbE anyway?
You could easily have said that nobody would want more than 128kb/s ISDN, because nobody does anything more intensive than download music, check e-mail, and watch flash animations. Faster speeds made newer services, services that are used by very ordinary people (like Netflix, Hangouts, Skype) possible.
Yet when you subject infrastructure spending to the political process, the best you can hope for is keeping up with present demand. Early adopters get no traction when they're proposing spending public money.
I want enough bandwidth to stream three HD 1080p movies at the same time (there are three people in my house) with enough left over for VOIP Phone and normal internet. I currently pay $80/month for that. ($50 for internet, 28+change for VOIP). I am willing to pay $120/month (that's frankly $4/day, not enough to really matter) for that. Can I buy it? No. Because no competition.
Why? Because utility monopoly. Why? Because lawyers and lobbyists and clueless and corrupt politicians.
Any question?
Why? Because utility monopoly. Why? Because lawyers and lobbyists and clueless and corrupt politicians.
Any question?
Cable companies are not utility monopolies. The granting of exclusive franchises has been illegal since 1992. Do they have lawyers and lobbyists? Sure. So do Google and Netflix and Facebook and Yahoo.
Companies don't give you what you want because: 1) its expensive; 2) they can make higher returns with that money elsewhere. You can blame lawyers and lobbyists all you want, but the bare fact is that what Facebook paid for WhatsApp would pay for all the lobbying capacity of the top 10 DC lobbying firms for 60 years. That's a really weak argument to lean on when the interested tech players have so much money and lobbying is so cheap.
Just look at what Google is doing with Google Fiber. They're demanding massive regulatory concessions, and still don't seem to be positioning it as a money-making business. If building fiber was a good use of capital, why would internet companies sit on the sidelines and demand someone else do it?
Companies don't give you what you want because: 1) its expensive; 2) they can make higher returns with that money elsewhere. You can blame lawyers and lobbyists all you want, but the bare fact is that what Facebook paid for WhatsApp would pay for all the lobbying capacity of the top 10 DC lobbying firms for 60 years. That's a really weak argument to lean on when the interested tech players have so much money and lobbying is so cheap.
Just look at what Google is doing with Google Fiber. They're demanding massive regulatory concessions, and still don't seem to be positioning it as a money-making business. If building fiber was a good use of capital, why would internet companies sit on the sidelines and demand someone else do it?
I appreciate your insight, but it really is still a political battle.
The reason why is because companies have to work with local governments to get construction permits to build out the access network. There are huge barriers here that prevent individuals and small companies from actually getting that work done. One of those barriers is the fact that even local governments have problems with bribery and corruption (you don't say!).
Google Fiber is demanding those massive regulatory concessions because they must! They're asking governments to bend over backwards for them because they recognize how hard it is go get through the bureaucracy. That's why they put on this big campaign to get the people to talk to their local representatives and essentially beg to be saved from the monopolies.
Take for example, Google Fiber versus Sonic.net. Sonic's been trying to build out their fiber network for a long time. However, since they're small it's much more difficult for them to work with the local governments and as a result their fiber rollout has been slower despite consumer demands.
The reason why is because companies have to work with local governments to get construction permits to build out the access network. There are huge barriers here that prevent individuals and small companies from actually getting that work done. One of those barriers is the fact that even local governments have problems with bribery and corruption (you don't say!).
Google Fiber is demanding those massive regulatory concessions because they must! They're asking governments to bend over backwards for them because they recognize how hard it is go get through the bureaucracy. That's why they put on this big campaign to get the people to talk to their local representatives and essentially beg to be saved from the monopolies.
Take for example, Google Fiber versus Sonic.net. Sonic's been trying to build out their fiber network for a long time. However, since they're small it's much more difficult for them to work with the local governments and as a result their fiber rollout has been slower despite consumer demands.
Please, show me the bribery and corruption that's creating these political hurdles. Otherwise, that's a strong accusation to be making without proof. If we're talking about good old fashioned lobbying, however, then the fact is that the internet companies have more than enough money to overwhelm the cable companies' lobbying efforts. Lobbyists are up for the highest bidder, after all. Yes, small companies can't afford it, but anyone with the money to realistically build the infrastructure in the first place can. Certainly, the companies with the most at stake (Google, Facebook, Netflix) can. It makes no sense to invoke money and lobbying while ignoring the fact of who has the deepest pockets in this game. Remember, Google is more profitable than Comcast and TWV combined, and has double the profit margin. Facebook's profits are about 2/3 as much as TWC.
The regulatory hurdles that Google is demanding relief from are (largely) not the result of lobbying. They're the result of the dysfunction of municipal politics. Right now, NYC's mayor is attacking Verizon because poor people can't afford FIOS (at $75/month). He's hired a civil rights lawyer to get into the issue. Is it any wonder companies aren't interested in building fiber? Is this the result of lobbying (or corruption and bribes) or predictable political forces?
At bottom, none of the screeds on this subject address the simple fact: the internet companies aren't rushing to build fiber, or lobbying to get permission to build fiber, or publicly demanding deregulation so they can build fiber. They're trying to get Comcast, Verizon, etc, to build fiber. To this day, Google positions Google Fiber as an effort to shame the ISPs, not a worthwhile business venture standing alone. What does that tell you about the monetary incentives at play?
The regulatory hurdles that Google is demanding relief from are (largely) not the result of lobbying. They're the result of the dysfunction of municipal politics. Right now, NYC's mayor is attacking Verizon because poor people can't afford FIOS (at $75/month). He's hired a civil rights lawyer to get into the issue. Is it any wonder companies aren't interested in building fiber? Is this the result of lobbying (or corruption and bribes) or predictable political forces?
At bottom, none of the screeds on this subject address the simple fact: the internet companies aren't rushing to build fiber, or lobbying to get permission to build fiber, or publicly demanding deregulation so they can build fiber. They're trying to get Comcast, Verizon, etc, to build fiber. To this day, Google positions Google Fiber as an effort to shame the ISPs, not a worthwhile business venture standing alone. What does that tell you about the monetary incentives at play?
For now.
In other words, who wants to spend billions on infrastructure they are guaranteed to profit from due to regulated rates set on a cost-plus basis?
And yet telcos weren't happy with the regulated profit that unbundled DSL provided them. Either they're just lying (which I consider entirely credible) or it's not that simple. For example, if the regulated profit is less than in an unregulated scenario, "shareholder value" thinking will argue for deregulation. Also, there's no profit if the regulated cost-plus price is so high that nobody buys it (what if the cost of FTTH is over $100/month?).
Who wants to spend billions on infrastructure they are forced to lease out at wholesale rates to competitors?
The government should, but they wouldn't lease them out to competitors, but providers who happen to be in competition with each other. Make the last mile a utility, like sewers.
The government should, but they wouldn't lease them out to competitors, but providers who happen to be in competition with each other. Make the last mile a utility, like sewers.
Lots of local community governments would love to do this. Unfortunately Comcast is working very hard to make sure that this is not allowed.
Right, but having more last mile networks would create pressure to resolve the connecting last mile ISP to infrastructure ISP issue.
Once the FCC finally takes the brakes off unlicensed wireless, rural and lower-density urban customers will see a competitive market of WISPs. They've been dragging it out for years, though, so who knows when that will happen?
Are existing WISPs spectrum-constrained today?
Thanks a lot for your response (and to the others who responded as well, appreciate it!). What you're saying makes sense, but just to play devils advocate...
Disclaimer: The blog I'm about to post was linked from Comcast's blog. They claim he is independent, I know nothing about him.
"Netflix could use multiple providers to connect to ISPs and could also use third party CDNs like Akamai, EdgeCast and Limelight, who are already connected to ISPs, to deliver their traffic. In fact, this is how Netflix delivered 100% of their traffic for many, many years, using third-party CDNs. Netflix likes to make it sound like there is only one way to deliver videos on the Internet when in fact, there are multiple ways."[1]
It would seem to me that if Comcast is using all the bandwidth they've purchased from Level 3 than it's a simple reality for them that they'll need to increase that bandwidth or they'll see reduced service. I don't see why this is explicitly either side's problem. It seems like it would be great if they could figure out a mutually beneficial situation, but I guess I fail to see the moral hazard. If Comcast doesn't want to upgrade their network who cares? Barring, of course, the monopoly argument, I see no moral hazard here.
I also see no reason that this has anything to do with net neutrality. It seems like Netflix is simply so big they have to act more like an ISP than they used to.
[1] http://blog.streamingmedia.com/2014/03/netflix-level-3-telli...
Disclaimer: The blog I'm about to post was linked from Comcast's blog. They claim he is independent, I know nothing about him.
"Netflix could use multiple providers to connect to ISPs and could also use third party CDNs like Akamai, EdgeCast and Limelight, who are already connected to ISPs, to deliver their traffic. In fact, this is how Netflix delivered 100% of their traffic for many, many years, using third-party CDNs. Netflix likes to make it sound like there is only one way to deliver videos on the Internet when in fact, there are multiple ways."[1]
It would seem to me that if Comcast is using all the bandwidth they've purchased from Level 3 than it's a simple reality for them that they'll need to increase that bandwidth or they'll see reduced service. I don't see why this is explicitly either side's problem. It seems like it would be great if they could figure out a mutually beneficial situation, but I guess I fail to see the moral hazard. If Comcast doesn't want to upgrade their network who cares? Barring, of course, the monopoly argument, I see no moral hazard here.
I also see no reason that this has anything to do with net neutrality. It seems like Netflix is simply so big they have to act more like an ISP than they used to.
[1] http://blog.streamingmedia.com/2014/03/netflix-level-3-telli...
> "It seems like it would be great if they could figure out a mutually beneficial situation, but I guess I fail to see the moral hazard."
This article in particular is making the argument that at the level that L3, Comcast, AT&T, Netflix and others are operating, it's almost always mutually beneficial to peer. The article also makes the point that the capital expenditure is negligible compared to the benefit to both sides.
The moral hazard here is that Comcast is (shrewdly) taking the bet that their customers won't blame them for what appears to be Netflix's reduced performance. If there was actual competition in the local ISP space, customers would catch on that the performance is only worse on Comcast's network, but that's not the world we live in.
> Barring, of course, the monopoly argument, I see no moral hazard here.
You can't just ignore the monopoly argument. There's nothing inherently unfair about a monopoly, but if you use it to extort money from people then its no surprise that people call you out on it. In the Good Old Days, you'd see monopolies try to squash their competitors. This is more nuanced than that. Monopolistic-extortion-action-at-a-distance, if you will.
This article in particular is making the argument that at the level that L3, Comcast, AT&T, Netflix and others are operating, it's almost always mutually beneficial to peer. The article also makes the point that the capital expenditure is negligible compared to the benefit to both sides.
The moral hazard here is that Comcast is (shrewdly) taking the bet that their customers won't blame them for what appears to be Netflix's reduced performance. If there was actual competition in the local ISP space, customers would catch on that the performance is only worse on Comcast's network, but that's not the world we live in.
> Barring, of course, the monopoly argument, I see no moral hazard here.
You can't just ignore the monopoly argument. There's nothing inherently unfair about a monopoly, but if you use it to extort money from people then its no surprise that people call you out on it. In the Good Old Days, you'd see monopolies try to squash their competitors. This is more nuanced than that. Monopolistic-extortion-action-at-a-distance, if you will.
> it's almost always mutually beneficial to peer.
But Netflix might be the exception here right?
> You can't just ignore the monopoly argument.
You're right, I'm just trying to disambiguate it from the net neutrality argument, which doesn't seem to actually be relevant here (as Comcast is treating all traffic through those links poorly, not just Netflix traffic). If it is truly about the monopoly, regulators have different (and thankfully more powerful) tools to change Comcast's behavior, whereas using net neutrality seems like a losing (and misleading) fight.
But Netflix might be the exception here right?
> You can't just ignore the monopoly argument.
You're right, I'm just trying to disambiguate it from the net neutrality argument, which doesn't seem to actually be relevant here (as Comcast is treating all traffic through those links poorly, not just Netflix traffic). If it is truly about the monopoly, regulators have different (and thankfully more powerful) tools to change Comcast's behavior, whereas using net neutrality seems like a losing (and misleading) fight.
> But Netflix might be the exception here right?
The mutual benefit still exists. Comcast benefits from peering with Netflix because it allows Comcast customers to access Netflix. Comcast is just playing chicken because they know their monopoly position allows them to hold out longer than peers with actual competitors.
> You're right, I'm just trying to disambiguate it from the net neutrality argument, which doesn't seem to actually be relevant here (as Comcast is treating all traffic through those links poorly, not just Netflix traffic).
The network neutrality issue is that Comcast is intentionally causing congestion on all the links that Netflix traffic could possibly use. The traffic for Comcast's own competing video service offerings don't have to traverse the congested links, nor does traffic from anyone who Comcast provides with a separate uncongested link in exchange for a toll or some other business consideration. That's practically the definition of a network neutrality issue.
The mutual benefit still exists. Comcast benefits from peering with Netflix because it allows Comcast customers to access Netflix. Comcast is just playing chicken because they know their monopoly position allows them to hold out longer than peers with actual competitors.
> You're right, I'm just trying to disambiguate it from the net neutrality argument, which doesn't seem to actually be relevant here (as Comcast is treating all traffic through those links poorly, not just Netflix traffic).
The network neutrality issue is that Comcast is intentionally causing congestion on all the links that Netflix traffic could possibly use. The traffic for Comcast's own competing video service offerings don't have to traverse the congested links, nor does traffic from anyone who Comcast provides with a separate uncongested link in exchange for a toll or some other business consideration. That's practically the definition of a network neutrality issue.
> intentionally causing congestion
Not upgrading their connection is not the same thing as intentionally causing congestion. Especially when the connection would be fine if it weren't for the massive share of Netflix traffic.
Not upgrading their connection is not the same thing as intentionally causing congestion. Especially when the connection would be fine if it weren't for the massive share of Netflix traffic.
The massive Netflix traffic that I, as a Comcast subscriber, is requesting.
If we're all all drawing too much data over the network, upgrade it and charge us more! I have no higher bandwidth options than Comcast's cable internet service. Huge lack of competition in California.
If we're all all drawing too much data over the network, upgrade it and charge us more! I have no higher bandwidth options than Comcast's cable internet service. Huge lack of competition in California.
Comcast wants to switch people from http://www.comcast.com/peering to http://www.comcast.com/dedicatedinternet/?SCRedirect=true
The former is what Level 3 does, the latter Netflix.
The former is what Level 3 does, the latter Netflix.
> But Netflix might be the exception here right?
Netflix is not infrastructure, they're not peers, they're Level 3 clients.
Netflix is not infrastructure, they're not peers, they're Level 3 clients.
Actually, that's no longer true under the deal Netflix recently worked out with Comcast:
http://www.washingtonpost.com/blogs/the-switch/wp/2014/02/23...
http://www.washingtonpost.com/blogs/the-switch/wp/2014/02/23...
If the ISP is letting a connection become saturated in order to place pressure on a third party provider to provide settlement, they are doing so at the cost of any other providers running over those trunks. Since switching to AT&T, I've noticed decreased Netflix performance and sometimes Imgur loads like I'm on dial-up again. I haven't checked the routes but it has me wondering if it's shared congestion. To your point, I'm not sure this is a moral hazard yet but it does seem like one of those things that could be very bad for the customer and the ISP can simply get away with it due to lack of competition.
I have two net connections, and both are generally reliable, but I've noticed that the DSL connection (over Windstream) is very slow for certain sites like imgur and tumblr and AWS, so I've set those to use the cable modem (Time Warner) connection, where they work great.
Since I have 2 connections I definitely have competition in my area. (Wireless is another option.) So I don't get why if a connection has issues, it's always the DSL one, and always with certain other sites.
Since I have 2 connections I definitely have competition in my area. (Wireless is another option.) So I don't get why if a connection has issues, it's always the DSL one, and always with certain other sites.
DSL + cable doesn't really count as competition, unless you can choose from multiple cable and multiple DSL providers.
Why doesn't it count?
Because the two have different physical limitations. DSL is strongly limited by distance to a cabinet, and most people can't get any better than 7mbit/s, 20 if you're lucky. Thus, DSL provides no pressure on cable, and the only reasons people use DSL are because they get it bundled with their phone and TV service, or because they got fed up with the cable company and switched to a lesser service out of frustration.
In areas where something actually competitive is available, cable companies rush to improve bandwidth, or offer two-year contract deals at undercut prices to try to outlive the competition, then jack the prices back up when the competition goes bankrupt.
In areas where something actually competitive is available, cable companies rush to improve bandwidth, or offer two-year contract deals at undercut prices to try to outlive the competition, then jack the prices back up when the competition goes bankrupt.
> only reasons people use DSL are because they get it bundled with their phone and TV service
Some people use DSL because they're happy with DSL. DSL can be had for cheaper, and many people do not need the higher speeds of cable. HN readers and other geeks who dominate these online discussions assume that everyone is like them and that they need the fastest speed possible while sneering at any available alternative that is "just" 7 megabits. 7 megabits gets you instant page loads on any website. Sure it't not as good if you want to stream four movies to your house at once...but not everyone needs that.
The fact is that Comcast is right when they point out that they do face competition, not only from DSL but also from other services such as wireless. That you and many other people do not like these alternatives does not mean they do not exist. Many people find them to be perfectly adequate and for them the other services are the best value.
Some people use DSL because they're happy with DSL. DSL can be had for cheaper, and many people do not need the higher speeds of cable. HN readers and other geeks who dominate these online discussions assume that everyone is like them and that they need the fastest speed possible while sneering at any available alternative that is "just" 7 megabits. 7 megabits gets you instant page loads on any website. Sure it't not as good if you want to stream four movies to your house at once...but not everyone needs that.
The fact is that Comcast is right when they point out that they do face competition, not only from DSL but also from other services such as wireless. That you and many other people do not like these alternatives does not mean they do not exist. Many people find them to be perfectly adequate and for them the other services are the best value.
OK, so crappy DSL doesn't count as competition. But U-Verse vs. cable is not "no competition".
Honestly, I've had one connection or the other completely fail, and (aside from the weird issues above) I sometimes haven't noticed for a few days, despite heavy Netflix streaming on 2 devices. Whatever is going on in my local market, to me it seems to be functioning well.
Couple of points: Comcast isn't buying bandwidth from Level 3. "Settlement-free peering" means no money is exchanged. The reason they don't charge each other is because it is mutually beneficial; Comcast gets to charge it's customers for access to content that passes through Level 3 and Level 3 gets to charge it's customers for access to the Comcast subscribers.
The problem is that there is only a market solution for the Level 3 customers' side; if Level 3 can't accept all the traffic that their customers are sending it, they have other options with who to connect to, as there are other backbone providers. It is in their financial interest to maintain their network capacity, or their clients will go elsewhere.
Comcast, on the other hand, has no market pressure. If they can't handle all the data that their customers are trying to access, their customers have no other option. They are stuck with Comcast.
And therein lies the rub; what both Level 3's and Comcast's customers want is access to each other. Neither company can provide that service on their own; Comcast doesn't have a global network that can send data around the world, while Level 3 doesn't have a last-mile network that connects to all the end users. However, only Level 3 has to worry about losing its customers to a competitor. Comcast can essentially hold all parties hostage; they can demand payments from every party: end users, Level 3, and Level 3's customers. Of course, holding everyone hostage hurts everyone, including their own customers; however, since their customers have no other options, they aren't really in danger of losing them. They can hold out longer than Level 3 can, because if Level 3 doesn't agree to pay Comcast's ransom, someone else might, and Level 3 will lose their customers to whoever pays Comcast's ransom.
The problem is that there is only a market solution for the Level 3 customers' side; if Level 3 can't accept all the traffic that their customers are sending it, they have other options with who to connect to, as there are other backbone providers. It is in their financial interest to maintain their network capacity, or their clients will go elsewhere.
Comcast, on the other hand, has no market pressure. If they can't handle all the data that their customers are trying to access, their customers have no other option. They are stuck with Comcast.
And therein lies the rub; what both Level 3's and Comcast's customers want is access to each other. Neither company can provide that service on their own; Comcast doesn't have a global network that can send data around the world, while Level 3 doesn't have a last-mile network that connects to all the end users. However, only Level 3 has to worry about losing its customers to a competitor. Comcast can essentially hold all parties hostage; they can demand payments from every party: end users, Level 3, and Level 3's customers. Of course, holding everyone hostage hurts everyone, including their own customers; however, since their customers have no other options, they aren't really in danger of losing them. They can hold out longer than Level 3 can, because if Level 3 doesn't agree to pay Comcast's ransom, someone else might, and Level 3 will lose their customers to whoever pays Comcast's ransom.
" Of course, holding everyone hostage hurts everyone, including their own customers;"
Everyone except Comcast.
Everyone except Comcast.
> I also see no reason that this has anything to do with net neutrality. It seems like Netflix is simply so big they have to act more like an ISP than they used to.
When Comcast let links to peers get saturated it is a network neutrality issue. Those services they provide will perform and competitors will not. Those who pay for access will also perform better than those who don't pay. It is by definition not neutral.
When Comcast let links to peers get saturated it is a network neutrality issue. Those services they provide will perform and competitors will not. Those who pay for access will also perform better than those who don't pay. It is by definition not neutral.
There's nothing wrong with the action of not delivering traffic, except that it goes against the spirit of their contract with their customers and the obvious damage it will do to capitalism if their customers have no option to leave for a better company.
There's a huge problem with the monopoly status of Comcast in this case: I'd dump an ISP for not quickly (ie, in a 3-4 month span) working something out with Netflix if I could.
Comcast's monopoly status is greatly impacting their negotiating position, and is core to this issue.
I mean, I'm trying to get a deal worked out with Netflix where they send me some information. For various technical reasons, this deal needs to go through a couple middlemen. Literally the only middleman I can hire is refusing to work out a deal with them that doesn't give an unusual benefit to that middleman (while Netflix's middleman is offering the standard package), and is screwing me in the process. I'd take literally any other middleman if I could, but this particular middleman has political connections, and stops that from being possible.
Comcast's monopoly status is greatly impacting their negotiating position, and is core to this issue.
I mean, I'm trying to get a deal worked out with Netflix where they send me some information. For various technical reasons, this deal needs to go through a couple middlemen. Literally the only middleman I can hire is refusing to work out a deal with them that doesn't give an unusual benefit to that middleman (while Netflix's middleman is offering the standard package), and is screwing me in the process. I'd take literally any other middleman if I could, but this particular middleman has political connections, and stops that from being possible.
> Comcast needs to do this because they're selling a connection to the "Internet" to their customers, not a connection to the servers Comcast owns, and to other Comcast customers.
Only, I'm sure, because it's untenable to do that. Yet. Cable execs really, really like their model of charging people for channels; it lets them extract maximum revenue. If they can do that with the Internet, they will. It may be difficult to do that on the consumer side yet, but what they're doing with Netflix is basically a complicated way to sneak more Comcast charges on to their customer's bills. In this case, via the Netfilx bill.
Only, I'm sure, because it's untenable to do that. Yet. Cable execs really, really like their model of charging people for channels; it lets them extract maximum revenue. If they can do that with the Internet, they will. It may be difficult to do that on the consumer side yet, but what they're doing with Netflix is basically a complicated way to sneak more Comcast charges on to their customer's bills. In this case, via the Netfilx bill.
You cannot pull anything from the Internet. Merely request that someone sends data to you.
EDIT: I understand it's an emotional issue but it's true. You can only send packets and hope others will talk to you.
EDIT: I understand it's an emotional issue but it's true. You can only send packets and hope others will talk to you.
You are downvoted not because of emotion, but because your comment is not germane to the discussion. The reason the GP phrased it the way he did is to make it clear whose volition is at play in the data transfer. The technical details at the TCP level couldn't matter less.
This would all be fine and good if Level3 didn't make money from selling bandwidth. Comcast doesn't have to buy a connection from Level3; Level3 will give Comcast a link for free because they know they can turn around and charge their own customers for access to it. Comcast is really just cutting out the middleman here; and that scares Level3.
They could only cut out the middle man if they built their own global network; they don't have any fiber running to Europe or Asia or elsewhere. If there were no middle man, Comcast customers could only reach other Comcast customers.
But they are cutting out the middleman with respect to large traffic sources. Netflix won't be the only one to pay for an interconnect; AWS, Google and Facebook will likely follow suit shortly. Take that business away from the transit providers and it hurts.
Someone is still going to have to send that data across the ocean and other long distances.
But they cant cut out the middleman. Comcast doesnt have a direct connection to netflix or anywhere else on the internet.
I think Level 3 should straight up shut off their peering agreement with comcast and watch how all the last mile customers blame comcast for the issue, until such time as comcast stops playing games.
I think Level 3 should straight up shut off their peering agreement with comcast and watch how all the last mile customers blame comcast for the issue, until such time as comcast stops playing games.
That assumes they blame Comcast. Maybe they blame Netflix for not working.
If you get to set the narrative such that one side or the other as the only ones who gain from the interconnection, this becomes trivial, but the hard facts are that both sides want the interconnection, but to try as hard to pretend that the other side is the more desperate one in the deal.
If you get to set the narrative such that one side or the other as the only ones who gain from the interconnection, this becomes trivial, but the hard facts are that both sides want the interconnection, but to try as hard to pretend that the other side is the more desperate one in the deal.
Comcasts customers pay Comcast for service. If every website they went to didnt work, they'd be calling, and blaming, Comcast.
> Comcast doesnt have a direct connection to netflix
They do now that a deal has been worked out:
http://www.washingtonpost.com/blogs/the-switch/wp/2014/02/23...
They do now that a deal has been worked out:
http://www.washingtonpost.com/blogs/the-switch/wp/2014/02/23...
All of their customers will still have to go through middlemen. Thats how the internet functions.
> All of their customers will still have to go through middlemen.
Not if they're watching Netflix. As I read the deal, Comcast and Netflix will be directly connected, so if you're a Comcast customer watching Netflix, your packets only traverse Comcast's and Netflix's networks; no third parties are involved.
Not if they're watching Netflix. As I read the deal, Comcast and Netflix will be directly connected, so if you're a Comcast customer watching Netflix, your packets only traverse Comcast's and Netflix's networks; no third parties are involved.
But the Internet doesn't work that way: if Level3 shut off their peering with Comcast, Level3's customers would just find another transit provider and route around L3 because those companies are still in business to make money. They don't care about squabbles between Comcast and L3; they just want their monetized traffic to get delivered. The last-mile customers probably wouldn't even notice; but Level3's customers would likely be forced to pay more for transit from another company. But they would still pay.
MSO revenue is threatened by alternate entertainment content, and is terrified of becoming a dumb pipe with no add-on fees. The issue is that Comcast wants to both connect to the internet for free & get paid by both the viewer and the content provider as they are in yesterday's cable model.
Comcast is connected to the internet through 'settlement-free' peering. Comcast can command this because they have grown large enough to trade access to their customer base for access to the transport network's data. Comcast would like to sell direct access to their network to the content providers, but would like to see more money for the service that the content providers are willing to pay.
Comcast has allowed (through inaction of normal peering upgrades) the interconnections to Level-3/Cogent to become saturated. The primary data on those interconnections is data that competes with the normal content that Comcast is used to delivering to their customers. Cogent and Level-3 offer less expensive bandwidth than their competitors. This is because of their lack of incumbent telephony models, and being dedicated fiber bit pushers.
Comcast is claiming that it is too expensive for them to deliver the service levels to their customers that their customers have contracted. Comcast is demanding that the companies providing the data (not the transit network delivering the data) must pay to have their traffic delivered the last mile. Simultaneously to crying that their internal networks are saturated and require upgrade, the MSOs are offering products that allow their customers to use the local provider's network (for locally sourced content) without incurring billing penalties or usage caps.
(veteran employee of a transit network involved)
Comcast is connected to the internet through 'settlement-free' peering. Comcast can command this because they have grown large enough to trade access to their customer base for access to the transport network's data. Comcast would like to sell direct access to their network to the content providers, but would like to see more money for the service that the content providers are willing to pay.
Comcast has allowed (through inaction of normal peering upgrades) the interconnections to Level-3/Cogent to become saturated. The primary data on those interconnections is data that competes with the normal content that Comcast is used to delivering to their customers. Cogent and Level-3 offer less expensive bandwidth than their competitors. This is because of their lack of incumbent telephony models, and being dedicated fiber bit pushers.
Comcast is claiming that it is too expensive for them to deliver the service levels to their customers that their customers have contracted. Comcast is demanding that the companies providing the data (not the transit network delivering the data) must pay to have their traffic delivered the last mile. Simultaneously to crying that their internal networks are saturated and require upgrade, the MSOs are offering products that allow their customers to use the local provider's network (for locally sourced content) without incurring billing penalties or usage caps.
(veteran employee of a transit network involved)
Comcast promises a certain amount of download capability. When customer requests for content that originate from Comcast's network (that would not normally exceed that download capability) end up saturating a link to which Comcast is party to, then it is Comcast's obligation to attempt to upgrade that link. Comcast cannot control the routes via which the content is returned, they must react to their customers usage patterns to improve the performance of their network. So if tomorrow, everyone wants to download content from some other peer then Comcast will need to upgrade other peer connections.
The exception to this is if Comcast wants to charge rent to the internet to have access to Comcast customers. Then it is in Comcast's corporate interest to let peer connections saturate and then charge upstream content providers rent to remove the congestion.
A pretty ballsy move for a company (who's assets are stretched out over thousands of miles and in plain sight) to attempt to extort the entire planet. ;)
The exception to this is if Comcast wants to charge rent to the internet to have access to Comcast customers. Then it is in Comcast's corporate interest to let peer connections saturate and then charge upstream content providers rent to remove the congestion.
A pretty ballsy move for a company (who's assets are stretched out over thousands of miles and in plain sight) to attempt to extort the entire planet. ;)
...who's assets are stretched out over thousands of miles and in plain sight...
I like the cut of your jib.
I like the cut of your jib.
Not very ballsy if they've bought enough politicians and lobbyists.
He means it's ballsy because any one of us could just start cutting their physical lines.
My understanding is this. No Money exchanges hands between Level3 and Comcast, but there is an agreement for Comcast to provide a port for Level3 to connect to. Level3 brings the traffic to Comcast door Step. Level3 pays for the port on their side. Level3 is happy to add more ports, BUT Comcast Refuses. Not because more ports are that expensive but those in charge at Comcast are from the TV side of the house where blackouts are common negotiating tactics in order to be paid. And Level3 refuse to be held over a barrel for payment when Comcast has a local monopoly power.
[deleted]
I think Level3's issue is that Comcast is unwilling to increase the capacity of their links without payment from Level3. If Level3 engages in settlement-free peering with Comcast and their network traffic graph looks anything like the 100gbit link in the article, I don't blame them: it's a 20:1 disparity in traffic in vs traffic out.
What's happening here is that Level3 and other transit providers are starting to see their industry be squeezed by the big ISPs. If you have 5 ISPs that serve 90% of the customers in the US, and 10 service providers that generate 90% of the bandwidth, why do transit providers even exist? I think they see their market share decreasing significantly as more of their customers follow in Netflix's footsteps, so they're trying to pile on and start a grass-roots outrage like Netflix did. Problem there is that most people haven't heard of Level3, so it makes it more difficult to get people behind them.
And you are completely right about the whole Netflix issue being Netflix and Level3/Cogent's problem. Level3 and Cogent have business models based on selling transit acquired through settlement-free peering. Settlement-free peering assumes that bandwidth usage is roughly symmetric: when it's not symmetric, the side sending more bandwidth has to pay. Level3's position here seems a bit hypocritical: if they had a customer that was routinely sending data at a 20:1 ratio, they would charge that customer for sending more bandwidth. But they're expecting the big ISPs not to do the same to them?
What's happening here is that Level3 and other transit providers are starting to see their industry be squeezed by the big ISPs. If you have 5 ISPs that serve 90% of the customers in the US, and 10 service providers that generate 90% of the bandwidth, why do transit providers even exist? I think they see their market share decreasing significantly as more of their customers follow in Netflix's footsteps, so they're trying to pile on and start a grass-roots outrage like Netflix did. Problem there is that most people haven't heard of Level3, so it makes it more difficult to get people behind them.
And you are completely right about the whole Netflix issue being Netflix and Level3/Cogent's problem. Level3 and Cogent have business models based on selling transit acquired through settlement-free peering. Settlement-free peering assumes that bandwidth usage is roughly symmetric: when it's not symmetric, the side sending more bandwidth has to pay. Level3's position here seems a bit hypocritical: if they had a customer that was routinely sending data at a 20:1 ratio, they would charge that customer for sending more bandwidth. But they're expecting the big ISPs not to do the same to them?
The big trouble is that we need settlement-free peering, or we'll have a very different sort of internet, or maybe none at all. Though it's hard for some to remember (or to believe), there were big consumer networks prior to the advent of consumer-oriented Internet. Compuserve and AOL were their own networks. And they were abysmal. Peering not only ruined that business model, it managed to co-opt those networks and assimilate them into the Internet proper.
If peering is untenable because it leads to outcomes like the one we have now (Netflix) where it is fundamentally unfair to one party (assuming that is the case) or the other, then we're all screwed. There might not be a fair solution that still manages to resemble the Internet.
In one alternative, the Comcasts and the AT&Ts are smited and no longer exist, and Level 3 goes into the consumer internet business, connecting things end-to-end. That's not something we could trust them with. Same if Comcast replaces Level 3.
In another, Netflix continues to pay the extortion (if it is that) which we have little doubt that Comcast will continue to ratchet up the price for. Or Comcast has to shoulder that burden alone (and can't charge customers extra for doing so, without people screaming "net neutrality!").
Or maybe services like Netflix just can't exist in such an environment. Also a bad outcome.
Or god help us, internet infrastructure is nationalized, and the same people who manage our roads and traffic lights take over.
Am I blowing this out of proportion?
If peering is untenable because it leads to outcomes like the one we have now (Netflix) where it is fundamentally unfair to one party (assuming that is the case) or the other, then we're all screwed. There might not be a fair solution that still manages to resemble the Internet.
In one alternative, the Comcasts and the AT&Ts are smited and no longer exist, and Level 3 goes into the consumer internet business, connecting things end-to-end. That's not something we could trust them with. Same if Comcast replaces Level 3.
In another, Netflix continues to pay the extortion (if it is that) which we have little doubt that Comcast will continue to ratchet up the price for. Or Comcast has to shoulder that burden alone (and can't charge customers extra for doing so, without people screaming "net neutrality!").
Or maybe services like Netflix just can't exist in such an environment. Also a bad outcome.
Or god help us, internet infrastructure is nationalized, and the same people who manage our roads and traffic lights take over.
Am I blowing this out of proportion?
You could have a situation where the government mandates net neutrality and at the same time compensates the likes of Comcast in some way (recognising their role as public utility and basically sanctioning their monopoly, but without taking over their operations entirely). You could have an anti-monopolistic legal action that splits Comcast but prevents Level3 from taking over / being present in that market. Etc etc...
There are many ways to skin an onion, and full-scale nationalisation is probably the least likely outcome (cash-strapped governments don't need another headache right now). In fact, the real problem here is that one player is leveraging a monopolistic position; remove that position, and the Free Market should start working its magic again.
There are many ways to skin an onion, and full-scale nationalisation is probably the least likely outcome (cash-strapped governments don't need another headache right now). In fact, the real problem here is that one player is leveraging a monopolistic position; remove that position, and the Free Market should start working its magic again.
Comcast already enjoys a physical government granted monopoly of the 'cable' access rights to their customers.
> time compensates the likes of Comcast in some way
Serious question, how would one find the exact sum with which Comcast should be compensated in the absence of a functional market?
Serious question, how would one find the exact sum with which Comcast should be compensated in the absence of a functional market?
Analyse the available facts and make a best guess, just like we do with private companies operating as regulated monopolies in e.g. water supply.
How about: a representative Congress mandates strong net neutrality rules and Comcast runs a 20% margin business instead of a ~83% margin business?
We tried that once with Bell Telephone. Their profits were dialed down a bit via regulation, and it was an oppressive, horrible, and un-innovative network. You weren't allowed to buy a telephone and plug it into the jack at home... you could only rent them from the company.
Thankfully, that company and environment are gone. Let's not recreate them.
Thankfully, that company and environment are gone. Let's not recreate them.
> Settlement-free peering assumes that bandwidth usage is roughly symmetric: when it's not symmetric, the side sending more bandwidth has to pay.
Does it assume that?
Since about the time of the web browser, we've had consumer-focused ISPs. Consumer traffic is mostly small requests for large replies. But as the article suggests, Level3 has a lot of settlement-free peering with consumer ISPs. My belief is that being settlement free isn't about equality of packets, but equality of demand. E.g., I have a lot of servers made to serve consumers; you have a lot of consumers wanting access to my servers; let's just split the costs.
In any case, given that those Comcast packets are mostly requesting all those Level3 packets, it seems much fairer to charge Comcast disproportionately, as its their customers who are creating the demand.
Does it assume that?
Since about the time of the web browser, we've had consumer-focused ISPs. Consumer traffic is mostly small requests for large replies. But as the article suggests, Level3 has a lot of settlement-free peering with consumer ISPs. My belief is that being settlement free isn't about equality of packets, but equality of demand. E.g., I have a lot of servers made to serve consumers; you have a lot of consumers wanting access to my servers; let's just split the costs.
In any case, given that those Comcast packets are mostly requesting all those Level3 packets, it seems much fairer to charge Comcast disproportionately, as its their customers who are creating the demand.
Right, there is a disparity — so, explain why it shouldn't instead be Comcast paying Level3?
Even if you want to privileged "sending" as being more charging worthy since you control what you send— in this case it's Comcast customers that have requested those bits, hosts connected to the level3 network are providing them.
Networks like comcast have built out imbalanced networks which likely wouldn't exist without their access to monopoly infrastructure. Practically anyone they peer with except other consumer broadband monopolists is going to be unbalanced.
Even if you want to privileged "sending" as being more charging worthy since you control what you send— in this case it's Comcast customers that have requested those bits, hosts connected to the level3 network are providing them.
Networks like comcast have built out imbalanced networks which likely wouldn't exist without their access to monopoly infrastructure. Practically anyone they peer with except other consumer broadband monopolists is going to be unbalanced.
"Settlement-free peering assumes that bandwidth usage is roughly symmetric: when it's not symmetric, the side sending more bandwidth has to pay."
I've never understood why this is - especially in a case like this, where the sending side is sending because (the customers of) the receiving side explicitly asked for it.
I've never understood why this is - especially in a case like this, where the sending side is sending because (the customers of) the receiving side explicitly asked for it.
Because the cost of a packet is borne by the receiver while the sender is the one monetizing it.
The assumption is that if you're sending traffic to a customer, you are being paid for that activity through subscription fees, ads, donations, etc. That packet costs the receiver some minuscule amount to process. Even though it was requested by the ultimate receiver, the ultimate sender is only sending the packet because they make money from doing so. The middlemen only pass on the packet because their costs are covered and they make a small amount of profit. This setup also ensures that if the packets aren't being monetized in some way (i.e. if it's a DDoS) that there's a financial incentive to stop them.
The assumption is that if you're sending traffic to a customer, you are being paid for that activity through subscription fees, ads, donations, etc. That packet costs the receiver some minuscule amount to process. Even though it was requested by the ultimate receiver, the ultimate sender is only sending the packet because they make money from doing so. The middlemen only pass on the packet because their costs are covered and they make a small amount of profit. This setup also ensures that if the packets aren't being monetized in some way (i.e. if it's a DDoS) that there's a financial incentive to stop them.
> you are being paid for that activity through subscription fees, ads, donations, etc.
Last time I checked ISPs where heavily monetizing packet receiving to the tune of 5-10x what netflix monetizes from me. For typically slow, shitty service bundled with cable TV or phone I do not want.
Last time I checked ISPs where heavily monetizing packet receiving to the tune of 5-10x what netflix monetizes from me. For typically slow, shitty service bundled with cable TV or phone I do not want.
... and it's probably pure coincidence that Netflix is in direct competition with 1/3 of that bundle, too.
Because the cost of a packet is borne by the receiver...
That's true for hot-potato routing, but CDNs generally use cold-potato routing.
As for monetization, both Netflix and Comcast are getting paid by their respective customers.
That's true for hot-potato routing, but CDNs generally use cold-potato routing.
As for monetization, both Netflix and Comcast are getting paid by their respective customers.
It's no less true for cold-potato routing; the source is just closer to the endpoint. And CDNs doing cold-potato routing happily pay for their transit because the service THEY make money off of is providing CDN services to their customers, who theoretically make money off the content they pay the CDNs to distribute.
What it comes down to is Comcast has no incentive to ensure its routes to various Internet transit providers are GOOD if they're not monetizing them. Why should Comcast be treated differently than any other transit provider and not allowed to monetize its transit services? Netflix wasn't paying Comcast, so Comcast had no responsibility to help Netflix make its own service better when there were other ways Netflix could have alleviated the bandwidth situation (e.g. buy transit through another company).
What it comes down to is Comcast has no incentive to ensure its routes to various Internet transit providers are GOOD if they're not monetizing them. Why should Comcast be treated differently than any other transit provider and not allowed to monetize its transit services? Netflix wasn't paying Comcast, so Comcast had no responsibility to help Netflix make its own service better when there were other ways Netflix could have alleviated the bandwidth situation (e.g. buy transit through another company).
Comcast has no incentive to ensure its routes to various Internet transit providers are GOOD if they're not monetizing them.
Customers pay Comcast with money to establish those links. If that's not "monetizing", what on earth is?
Customers pay Comcast with money to establish those links. If that's not "monetizing", what on earth is?
The original L3 article actually touches on this, sort of, by mentioning markets where ISP port congestion is not an issue (i.e. the UK) due to competition between ISPs.
Comcast will get paid by their customers as long as the routes they pass traffic through are 'good enough'. They have a (near) monopoly on the last mile in many markets, so 'good enough' can mean both 'barely working' and 'better than any other option you've got' at the same time.
So, you're right, but so is the parent: Comcast has no financial incentive to provide any more than just enough bandwidth to keep you from calling the support line to complain or cancelling your contract in frustration.
Where's the rest of customer revenue going, if not towards network upgrades? Well, let's just say the Comcast Center (http://archrecord.construction.com/projects/bts/archives/off...) ain't paying for itself...
Comcast will get paid by their customers as long as the routes they pass traffic through are 'good enough'. They have a (near) monopoly on the last mile in many markets, so 'good enough' can mean both 'barely working' and 'better than any other option you've got' at the same time.
So, you're right, but so is the parent: Comcast has no financial incentive to provide any more than just enough bandwidth to keep you from calling the support line to complain or cancelling your contract in frustration.
Where's the rest of customer revenue going, if not towards network upgrades? Well, let's just say the Comcast Center (http://archrecord.construction.com/projects/bts/archives/off...) ain't paying for itself...
As others have said, Comcast is also being monetarily rewarded here. I think the best argument lies along "the sender is in the best position to take measures to reduce traffic (traded off against their goals)."
They are rewarded for providing access; not for passing individual packets. There's a difference.
How precisely are they providing access if they do not pass individual packets?
Edited to add: It's certainly true they are not rewarded for the marginal packet, but then neither is Netflix (given their subscription based model).
Edited to add: It's certainly true they are not rewarded for the marginal packet, but then neither is Netflix (given their subscription based model).
Level-3 is actually going around to their peers and trying to create systems to measure the bit-miles traversed on traffic sent to other tier-1 peers; the carriers will then pay for out-of-sync ratios
This "out vs in" comparison in order to determine who pays is a meme that won't die. I've never known agreements to be formed in such a way. Maybe before my day in the dial up era when "out vs in" could serve as rule of thumb of sorts to determin who was an access ISP and who was an "Upstream" or "Tier 1" ISP. Can someone provide a source for this?
Historically, ratios were used for peering between ISPs of the same type. When one backbone peers with another backbone, 1:1 ratio can be a simple proxy for equal value. (ISPs liked simplicity in the old days because it made the Internet cheaper than the complex telco X.25/ATM/SONET networks with their complex tariffs and settlement.) When a content provider peers with a broadband ISP, they can exchange equal value without exchanging equal traffic.
Torrenting provided a lot of traffic that could bring symmetry to the amounts send/received. And yet it was frowned upon.
Indeed: perhaps the answer to the symmetry argument is for Netflix clients to just start uploading scads of random data while the movie is streaming. Then the traffic will be "symmetric", although Comcast won't like it after all the last-mile upload links are saturated and customers can't do anything online.
Netflix hasn't done this (and likely won't) because would make it difficult/impossible to control the quality of their service. P2P connections are a bit of a crap-shoot based on things like router configuration, device type and resources. For example, P2P wouldn't work as well on an iPad where you're only caching a few hundred megs of video (and not the whole video) at any given time.
I don't think the parent meant to imply a P2P implementation of Netflix streaming video, just that Netflix is perfectly capable of making that ratio balance out by uploading crap from the clients they have (partial) control over - which would then be traffic coming from Comcast's network.
Thank you, I was having trouble imagining how I could have been that unclear.
Torrenting moves large amounts of traffic into last mile infrastructure which tends to already be congested because its very costly to upgrade.
Due to using many distinct flows it also tends to not be very TCP friendly and takes a fairly unequal share of the bandwidth. And while users will give up and do something else when their interactive service becomes slow, a congested torrenting host keeps torrenting.
Due to using many distinct flows it also tends to not be very TCP friendly and takes a fairly unequal share of the bandwidth. And while users will give up and do something else when their interactive service becomes slow, a congested torrenting host keeps torrenting.
There doesn't seem to be any congestion on the last mile infrastructure of providers like Comcast. Their problems are all with interconnection to the real internet. It would actually be better for Comcast if their customers were getting more content from eachother over p2p networks than from external providers like Netflix.
> it's a 20:1 disparity in traffic in vs traffic out.
Are you talking about on this graph?
http://blog.level3.com/wp-content/uploads/2014/05/route_info...
Because all the numbers I see are showing about a 5:1 or 6:1 imbalance.
> ...transit providers are starting to see their industry be squeezed by the big ISPs.
No.
> ...why do transit providers even exist?
Because it's untenable (and inefficient) for Comcast to build a separate fiber network to every service provider (e.g. to start providing a service on your version of the internet, you'd have to build your own network connection to Comcast, and then again to Verizon, and again to AT&T, and again to every other ISP on the planet).
Take another look at the L3 network map from the blog post:
http://blog.level3.com/wp-content/uploads/2014/05/network_ma...
If you think transit providers are purely middlemen for the sake of middlemen then you're basically saying that every service provider should build out a network of that same size in order to reach every ISP.
> Settlement-free peering assumes that bandwidth usage is roughly symmetric
No. It assumes that the benefit to each network is roughly symmetric. Put another way, if Netflix videos only degrade on Comcast's network, Comcast's goodwill with its own customers will increase as a result of increasing transit/peering capacity. To a normal company, that's a tangible benefit to the agreement enough to make it worthwhile.
Comcast empirically does not care about the goodwill of its customers, which is not surprising given its monopoly of local markets. If customers had alternatives to Comcast and it became known that Netflix worked on one ISP and not the other, customers would switch ISPs en masse.
Are you talking about on this graph?
http://blog.level3.com/wp-content/uploads/2014/05/route_info...
Because all the numbers I see are showing about a 5:1 or 6:1 imbalance.
> ...transit providers are starting to see their industry be squeezed by the big ISPs.
No.
> ...why do transit providers even exist?
Because it's untenable (and inefficient) for Comcast to build a separate fiber network to every service provider (e.g. to start providing a service on your version of the internet, you'd have to build your own network connection to Comcast, and then again to Verizon, and again to AT&T, and again to every other ISP on the planet).
Take another look at the L3 network map from the blog post:
http://blog.level3.com/wp-content/uploads/2014/05/network_ma...
If you think transit providers are purely middlemen for the sake of middlemen then you're basically saying that every service provider should build out a network of that same size in order to reach every ISP.
> Settlement-free peering assumes that bandwidth usage is roughly symmetric
No. It assumes that the benefit to each network is roughly symmetric. Put another way, if Netflix videos only degrade on Comcast's network, Comcast's goodwill with its own customers will increase as a result of increasing transit/peering capacity. To a normal company, that's a tangible benefit to the agreement enough to make it worthwhile.
Comcast empirically does not care about the goodwill of its customers, which is not surprising given its monopoly of local markets. If customers had alternatives to Comcast and it became known that Netflix worked on one ISP and not the other, customers would switch ISPs en masse.
I was going from memory on the 20:1 thing and I was wrong, but 5:1 is still not even close to symmetric.
I also used a bit of hyperbole in the "why do transit providers even exist" part; transit providers do still need to exist to provide access for the long tail. But Level3 and the like do see their market shrinking as the big fish who currently pay them for transit either directly or indirectly move more bandwidth to direct connections with the big ISPs. It is definitely starting to squeeze them.
As far as the benefit to each network goes, that's hogwash. It's not how the contracts are written because you can't quantify "benefit". You can quantify packets sent/received. Netflix needs to make money just as badly as Comcast does, so the fact that their service is crappy is much more their problem than it is Comcast's. Netflix can always purchase transit through another company that's not Comcast.
I also used a bit of hyperbole in the "why do transit providers even exist" part; transit providers do still need to exist to provide access for the long tail. But Level3 and the like do see their market shrinking as the big fish who currently pay them for transit either directly or indirectly move more bandwidth to direct connections with the big ISPs. It is definitely starting to squeeze them.
As far as the benefit to each network goes, that's hogwash. It's not how the contracts are written because you can't quantify "benefit". You can quantify packets sent/received. Netflix needs to make money just as badly as Comcast does, so the fact that their service is crappy is much more their problem than it is Comcast's. Netflix can always purchase transit through another company that's not Comcast.
I can assure you that L3 and Cogent are not seeing the marketplace shrink. 10k quarterly files from each discuss their growths and vulnerabilities.
Historically, eyeball networks have paid for transit from backbone providers. As market consolidation occurred, the cable providers with government granted physical monopolies were able to negotiate for better settlements with their transit providers.
Eventually, the cable companies condensed with the major telephone companies. Some of the new mega companies have backbones and can use their own networks for transit (Verizon, AT&T, ...) others (Comcast) were able to use their size and access to their customers to negotiate largely settlement free exchanges.
However it is important to note that ALL eyeball networks have 1:5 to 1:20 demand ratios. This is the nature of content versus consumption. There is no new news regarding these ratios, and they are not particularly germane unless attempting to engineer the flows.
The real and pertinent issue is that Comcast has not lived up to the 'timely upgrade' clauses in boilerplate (NDA'd) settlement-free peering agreements. Clearly they have an advantage to "defect" from the standard cooperation model. Netflix has chosen to change providers several times, and recently provided data on whom they pay for service.
Historically, eyeball networks have paid for transit from backbone providers. As market consolidation occurred, the cable providers with government granted physical monopolies were able to negotiate for better settlements with their transit providers.
Eventually, the cable companies condensed with the major telephone companies. Some of the new mega companies have backbones and can use their own networks for transit (Verizon, AT&T, ...) others (Comcast) were able to use their size and access to their customers to negotiate largely settlement free exchanges.
However it is important to note that ALL eyeball networks have 1:5 to 1:20 demand ratios. This is the nature of content versus consumption. There is no new news regarding these ratios, and they are not particularly germane unless attempting to engineer the flows.
The real and pertinent issue is that Comcast has not lived up to the 'timely upgrade' clauses in boilerplate (NDA'd) settlement-free peering agreements. Clearly they have an advantage to "defect" from the standard cooperation model. Netflix has chosen to change providers several times, and recently provided data on whom they pay for service.
Why should an end-user ISP ever expect a settlement-free peering bandwidth graph to be symmetric? The important question is whether customers are requesting connections that route through through a particular transit provider. Comcast gets to sell "Internet" access, while the transit provider gets to sell bandwidth. Win win, no symmetry needed.
...you can't quantify "benefit"....
Isn't that the entire purpose of an economy?
...you can't quantify "benefit"....
Isn't that the entire purpose of an economy?
> To a normal company, that's a tangible benefit to the agreement enough to make it worthwhile.
Can you qualify this a bit? What's the actual expect cost to Comcast here to fix the issue? General ballpark?
Can you qualify this a bit? What's the actual expect cost to Comcast here to fix the issue? General ballpark?
You're right to call me out here. Although I have a reasonable idea what it would take technically, I don't actually know the financial impact to Comcast for increasing capacity to L3 or Cogent (not even ballpark). However, I'm taking the L3 article at face value when it says:
But there are also typically shared costs for networks to
interconnect. Each party pays to augment its own network
to allow for more traffic exchange (the expense to augment
capacity is not significant for either party). And since
we often choose to interconnect in a third party data
center, the networks usually agree to share the cost of
the cross connects by paying for them on an alternating
basis.
Cogent's CEO has already offered to pay outright for the port costs and data-center cross-connects for any upgrades. The point is to prove that the infrastructure cost of the interconnect are not the issue.
Upgrading that interconnect mentioned in the original article seems like just plugging in some $10k-$20k standard hardware and connecting a single extra wire. That kind cost doesn't matter really, it seems that they're hurting their interconnects intentionally for political reasons.
If this is a game of chicken, Level 3 and Cogent should play it to the end game. They are powerful enough to get Comcast and the other ISPs to concede. Just null route any interconnect that consistently drops packets and fails to get negotiated for an upgrade. The catch? It must be done by both L3 and Cogent.
Comcast is playing the monopoly card against what is an oligopoly of backbone providers. Comcast isn't strategically as well positioned as their actions would indicate.
It is one thing for Comcast to have degraded Netflix service. It is a different thing to present their customers with access to only half of the Internet .
Comcast is playing the monopoly card against what is an oligopoly of backbone providers. Comcast isn't strategically as well positioned as their actions would indicate.
It is one thing for Comcast to have degraded Netflix service. It is a different thing to present their customers with access to only half of the Internet .
Except that it would probably violate their contracts to just stop services at an interconnect. It's not like Level 3 is losing customers or money from the congested interconnects. Their incentive to play hardball just isn't there.
Level3 and Cogent both bill the majority (by traffic volume) of their network consumers by utilization. This means that these congested pipes are actually costing potential revenue.
Obviously Cogent would be willing to cut off their own nose to spite their face, but Level3 plays a much more sane game.
Hardball by coordinating efforts of the backbone providers is somewhat thwarted by the coordinated efforts of the MSO's, some with their own backbones (VZ, AT&T). Cutting off Comcast would also invalidate a number of (government) customer contracts that require service 'to the full Internet'.
Obviously Cogent would be willing to cut off their own nose to spite their face, but Level3 plays a much more sane game.
Hardball by coordinating efforts of the backbone providers is somewhat thwarted by the coordinated efforts of the MSO's, some with their own backbones (VZ, AT&T). Cutting off Comcast would also invalidate a number of (government) customer contracts that require service 'to the full Internet'.
If it violates their contract, doesn't a soft fail violate it also? Or will violate, whenever the rate of dropped packets goes above a given threshold. I view the hard fail as just speeding up the interconnect failure, with the positive effect of prompting action from the last mile ISP.
Whenever their performance drops towards part of the internet, their value proposition gets lower. You can view the value chain as
Level 3 and equivalents, by cutting off non-performant interconnects will hurt their own customers, of course. However, they are just speeding up the same effect being applied softly over time, by Comcast and equivalents. A softly failing interconnect moves revenue over to last mile ISPs, thus permanently lowering Level 3's value proposition. By failing the interconnect sharply, instead of softly, they actually improve the steady state of the system, in general (good side effect), and in particular (the real objective).
Whenever their performance drops towards part of the internet, their value proposition gets lower. You can view the value chain as
(residential customer) <-> Comcast <-> Level 3 <-> (hosting customer)
Level 3 and equivalents, by cutting off non-performant interconnects will hurt their own customers, of course. However, they are just speeding up the same effect being applied softly over time, by Comcast and equivalents. A softly failing interconnect moves revenue over to last mile ISPs, thus permanently lowering Level 3's value proposition. By failing the interconnect sharply, instead of softly, they actually improve the steady state of the system, in general (good side effect), and in particular (the real objective).
I don't understand what's so hard about "You can roll with us as long as you treat our rented infrastructure with respect." That sort of agreement exists in many industries. I get kicked out of my apartment if I trash it, even though I paid for it. I don't ask my landlord to split the costs of an upgrade to hardwood floors because I fucked up the linoleum (because "I ended up with more pets than I expected" or something), much less pay for it himself. I pay for it or I'm booted. If he did anything else, such as let destructive renters stay on his property or paid for their damage they were responsible for, what would be the point of renting the property? That's a terrible business model. Same thing here... Comcast is damaging Level3's customer relationships on purpose and refusing to pay reasonable level-of-service costs to rectify the problem and should be thrown off the network.
I fail to see how you find someone like Level3 or Cogent more powerful than Comcast. I'd like to understand your reasoning behind such a position. Comcast is insanely powerful: somewhat because of the eyeballs they have on their network(s) and mostly because of their regulatory/political leverage and influence.
Summary of situation
Comcast cripples customers internet services to extort money from companies connecting Comcast to the internet for free.
Comcast cripples customers internet services to extort money from companies connecting Comcast to the internet for free.
It's completely crazy and just blatant extortion on Comcast's part.
The current system is pretty straight forward.
Level 3: Server(Who Level 3 Charges) -> Level 3 -> Comcast -> End User
Comcast: End User(Who Comcast Charges) -> Comcast -> Level 3 -> Server
Now Comcast wants to charge everyone in the chain that isn't them. They are attempting to do this by degrading (or at the very least not upgrading) their services for their customer.
The techno-libertarians that try to defend this nonsense are going to sink the whole ship and make every pay to receive even less.
The current system is pretty straight forward.
Level 3: Server(Who Level 3 Charges) -> Level 3 -> Comcast -> End User
Comcast: End User(Who Comcast Charges) -> Comcast -> Level 3 -> Server
Now Comcast wants to charge everyone in the chain that isn't them. They are attempting to do this by degrading (or at the very least not upgrading) their services for their customer.
The techno-libertarians that try to defend this nonsense are going to sink the whole ship and make every pay to receive even less.
Ahem. "Techno-libertarians" aren't the ones who gave Comcast its monopolies on the last mile. Please leave them out of this.
No, physics and economics did. When you're talking about building infrastructure that will take 20 years to pay itself off _absent_ competition, only the most profitable of the profitable locations can afford extensive physical plant competition.
In communications networks monopolies are natural, especially in last mile networks, and will exist absent any government action. That in some cases some governments have mildly strengthened these monopolies in exchange for commitments to cover otherwise unprofitable areas, its not really that important this subject.
In communications networks monopolies are natural, especially in last mile networks, and will exist absent any government action. That in some cases some governments have mildly strengthened these monopolies in exchange for commitments to cover otherwise unprofitable areas, its not really that important this subject.
They have been a strong voice for less regulation, especially less regulation of technology. We reined in AT&T through regulation, forcing them to share their last-mile plant with others. We could (and should) have done the same with monopoly cable providers, but the anti-regulation fever in the US prevented that.
Does anyone have an idea what the European ISP is that refuses to provide enough peering capacity?
I would have guessed it was French ISP Proxad/Free, which is notorious for having generally shitty peering to a large part of Internet (like Youtube, Imgur or Github, among others).
But though they are a large ISP they are not dominant, and their peering with Level3 is actually quite good... which is why I tunnel most of my connection to my server (the route to which happens to be through Level3) so I can get usable internet.
It really sucks to be able to download an iso from some server in the US at 2 MB/s, but seeing small Imgur gifs load frame by frame, or Github repos cloning at 20 kB/s.
But though they are a large ISP they are not dominant, and their peering with Level3 is actually quite good... which is why I tunnel most of my connection to my server (the route to which happens to be through Level3) so I can get usable internet.
It really sucks to be able to download an iso from some server in the US at 2 MB/s, but seeing small Imgur gifs load frame by frame, or Github repos cloning at 20 kB/s.
Free just doesn't buy enough transit; they are not a tier 1 provider.
The ISPs in question explicitly aren't tier 1 providers.
Free is tier 2, and they are very much behaving like the ISPs the article is talking about, just not with Level 3.
Free is tier 2, and they are very much behaving like the ISPs the article is talking about, just not with Level 3.
Free buys transit, and that transit is full. All the time. The article is talking about settlement free peering. There are significant differences between Free running their uplinks hot and a peering partners running their interconnects hot.
They said its not from the UK and judging from their connection points, I guess its Deutsche Telekom.
Don't know they described them as a monopoly position and cable is pretty strong in Germany. I think Spain or Italy might be better guesses.
Telefonica in Spain might be a reasonable guess. Ono the cable company isn't huge.
I think Telecom Italian is dominant in Italy and there is virtually no cable at all so that would be my guess.
Telefonica in Spain might be a reasonable guess. Ono the cable company isn't huge.
I think Telecom Italian is dominant in Italy and there is virtually no cable at all so that would be my guess.
Too late to fix typo: Telecom Italia (not Italian).
Yes, it seems it's the "Telekom" from Germany:
* http://www.golem.de/news/level-3-sechs-grosse-internetprovid...
* http://www.welt.de/wirtschaft/article117963991/Bei-Europas-T...
* http://www.golem.de/news/level-3-sechs-grosse-internetprovid...
* http://www.welt.de/wirtschaft/article117963991/Bei-Europas-T...
France Telecom
Level 3's filing with the FCC about this: http://www.level3.com/~/media/Assets/legal/l3_openinternet_m...
If Comcast doesn't like the amount of bandwidth their customers use watching Netflix, they always have the option to notify their customers and null-route or otherwise stop Netflix from working at all. However while it "works" but sucks they appear to be doing their jobs while their customers blame Netflix for sucking while largely being unaware that it's actually Comcast that sucks.
[deleted]
If you are experiencing a slow connection and packet loss, what is the best tool for identifying the issue? There must be a good way to find out where the problem is.
[deleted]
That doesn't seem right to me.