Facebook Operations Chief Reveals Open Networking Plan(enterprisetech.com)
enterprisetech.com
Facebook Operations Chief Reveals Open Networking Plan
http://www.enterprisetech.com/2013/10/04/facebook-operations-chief-reveals-open-networking-plan/
6 comments
It sounds like he was insinuating that a BGP based Solution scales better than a VLAN/Spanning Tree Solution when your infrastructure reaches an extremely large capacity. I'm not sure exactly what he's referring to, but I would guess that it has something to do with either the 12-bit VLAN ID limitation (802.1q maximum theoretical limit of 4096 VLAN ID's), or more likely a limitation relating to the MAC Address Table and/or a Solution that relies on MAC Addresses (for example, 802.1d spanning tree). I believe there are new approaches to overcome these limitations (for example, 802.1ab double tagging or VMWare's VXLAN), but I guess they just decided to skip Layer 2 entirely. I've never worked in an environment where BGP is used extensively internally, but I guess from a conceptual standpoint it could be done with a good ibgp/ebgp strategy and use of techniques like Route Reflectors and/or Confederations. Going with the Layer 3 approach also seems to match well with their vision of the Open Compute Network, where the Networking Appliances are viewed more like Virtual Appliances. Very cool stuff. Wish I worked there, way more fun that the stuff I'm working with.
Here are some things:
One scale problem with layer 2 networks is you have to go to layer 3 at some point and you have an aggregation node(s) that have to do a considerable amount of ARP request/reply traffic. The request/reply traffic is handled by the CPU of the machine processing. This can run the CPU hotter than you really want and if the aggregation nodes are control plane policing ARP (most sw vendors do), they will intermittently drop arp requests.
Once the request/reply is processed, the node pushes the ARP entry into hardware. ARP hardware tables are limited resources as well. If you exceed them, traffic will either be black-holed or forwarded by the CPU of the node. Both of these are undesirable.
Additionally, it becomes difficult to scale forwarding capacity of layer 2 spanning tree (or loop free) based networks to greater than 2 aggregation nodes since you will either block ports or create loops.
One scale problem with layer 2 networks is you have to go to layer 3 at some point and you have an aggregation node(s) that have to do a considerable amount of ARP request/reply traffic. The request/reply traffic is handled by the CPU of the machine processing. This can run the CPU hotter than you really want and if the aggregation nodes are control plane policing ARP (most sw vendors do), they will intermittently drop arp requests.
Once the request/reply is processed, the node pushes the ARP entry into hardware. ARP hardware tables are limited resources as well. If you exceed them, traffic will either be black-holed or forwarded by the CPU of the node. Both of these are undesirable.
Additionally, it becomes difficult to scale forwarding capacity of layer 2 spanning tree (or loop free) based networks to greater than 2 aggregation nodes since you will either block ports or create loops.
Hi pm90, I work on the facebook network engineering team, I can explain at a high-level some of the reasons we made this change.
You're correct BGP was originally intended for inter-AS routing, however, those AS's don't necessarily have to be different ISPs. They could be different divisions in a company, or different racks in a data-center. They can be any grouping where you might want different policies applied. You can also use BGP between devices in the same AS where a common policy makes sense. There's no technical limitation stopping you from using BGP inside the data-center, so it comes down to pros and cons of the various options out there.
Our main goals were scalability, reliability and simplicity. When it comes to scalability BGP is essentially the best, this is because, and why it's used between AS's where the routing tables can grow quite large. When it comes to reliability, we previously had a layer2 architecture, but there are a lot of challenges with that at scale, some examples:
* Split brain: In a layer2 world, you still typically have a layer3 protocol running between your upper layers. We were using BGP and VRRP at the time, and issues in one wouldn't always be communicated to the other, in some cases that's not possible. So VRRP may think everything is fine, while BGP is down, the result is black-holed trafic.
* Scale: Running an all layer2 network requires devices that have large enough CAM tables to support all connected devices, many vendors newer full line rate cards were coming out with smaller CAM tables, as such layer2 simply wasn't an option in some cases. Additionally, many of the other protocols that were options are computationally expensive to deploy at a large size, you either end up slowing them down, or deploying relatively complex designs.
One of the many significant benefits BGP brought us came in the form of ECMP. In a layer2/VRRP world, balancing traffic is hacky with two destinations, if you want to do more, which we did, it becomes increasingly complex. With BGP it's built in, we can consistently and easily balance the load to 8, 16, 32 or more destinations depending on the hardware and software in use.
BGP is also readily available in the software from the major network vendors, as well as within OpenSource solutions for the servers. This means we can use a common protocol everywhere. Which in turns makes sourcing and deploying new equipment much easier and simplifies the configs, policies and tooling needed to support the deployment.
There are many other reasons that layer2 wasn't a good choice for us, and that layer3 makes a lot of sense. I'd be happy to discuss more of these as well.
Hopefully this helps to answer your question.
You're correct BGP was originally intended for inter-AS routing, however, those AS's don't necessarily have to be different ISPs. They could be different divisions in a company, or different racks in a data-center. They can be any grouping where you might want different policies applied. You can also use BGP between devices in the same AS where a common policy makes sense. There's no technical limitation stopping you from using BGP inside the data-center, so it comes down to pros and cons of the various options out there.
Our main goals were scalability, reliability and simplicity. When it comes to scalability BGP is essentially the best, this is because, and why it's used between AS's where the routing tables can grow quite large. When it comes to reliability, we previously had a layer2 architecture, but there are a lot of challenges with that at scale, some examples:
* Split brain: In a layer2 world, you still typically have a layer3 protocol running between your upper layers. We were using BGP and VRRP at the time, and issues in one wouldn't always be communicated to the other, in some cases that's not possible. So VRRP may think everything is fine, while BGP is down, the result is black-holed trafic.
* Scale: Running an all layer2 network requires devices that have large enough CAM tables to support all connected devices, many vendors newer full line rate cards were coming out with smaller CAM tables, as such layer2 simply wasn't an option in some cases. Additionally, many of the other protocols that were options are computationally expensive to deploy at a large size, you either end up slowing them down, or deploying relatively complex designs.
One of the many significant benefits BGP brought us came in the form of ECMP. In a layer2/VRRP world, balancing traffic is hacky with two destinations, if you want to do more, which we did, it becomes increasingly complex. With BGP it's built in, we can consistently and easily balance the load to 8, 16, 32 or more destinations depending on the hardware and software in use.
BGP is also readily available in the software from the major network vendors, as well as within OpenSource solutions for the servers. This means we can use a common protocol everywhere. Which in turns makes sourcing and deploying new equipment much easier and simplifies the configs, policies and tooling needed to support the deployment.
There are many other reasons that layer2 wasn't a good choice for us, and that layer3 makes a lot of sense. I'd be happy to discuss more of these as well.
Hopefully this helps to answer your question.
>Running an all layer2 network requires devices that have large enough CAM tables to support all connected devices, many vendors newer full line rate cards were coming out with smaller CAM tables, as such layer2 simply wasn't an option in some cases.
Does this imply that mac forwarding tables(on switch) and arp cache(on hosts) need to have entries only for their immediate neighbours ?
Curious to know how much modified the host network stack is. Also, how do you provision a new server with the right IP ? Is this mechanism in L2 ?
>There are many other reasons that layer2 wasn't a good choice for us, and that layer3 makes a lot of sense. I'd be happy to discuss more of these as well.
I am sure people would find that very useful. Thanks for the excellent writeup !
Does this imply that mac forwarding tables(on switch) and arp cache(on hosts) need to have entries only for their immediate neighbours ?
Curious to know how much modified the host network stack is. Also, how do you provision a new server with the right IP ? Is this mechanism in L2 ?
>There are many other reasons that layer2 wasn't a good choice for us, and that layer3 makes a lot of sense. I'd be happy to discuss more of these as well.
I am sure people would find that very useful. Thanks for the excellent writeup !
The last-hop switches facing the servers are layer 3 in the direction of the aggregation switches and layer 2 in the direction of the servers so ARP is handled normally and DHCP can be handled with standard DHCP-helper mechanisms. This allows the host networking stack to be vanilla (if you want it to be).
> Scale: Running an all layer2 network requires devices that have large enough CAM tables to support all connected devices
... in the broadcast domain. Maybe you save on CAM, but now youre buying more TCAM/SRAM for your route table. You're going to get what, a couple thousand prefixes on your trident boxes? And burn a bunch on all the /32s and ECMP'd routes you have?
I'm quite curious why more routing looked better than label based switching in a meshed network.
... in the broadcast domain. Maybe you save on CAM, but now youre buying more TCAM/SRAM for your route table. You're going to get what, a couple thousand prefixes on your trident boxes? And burn a bunch on all the /32s and ECMP'd routes you have?
I'm quite curious why more routing looked better than label based switching in a meshed network.
To run BGP, you would not have to run /32s everywhere. You could have multiple tiers of aggregation that keep your routing tables manageable. It would look something like this.
hosts (/32) -> rack-sw -> (/26) -> aggregation-sw -> (/17)
If you need some specifics to be advertised, scale would be dependent on the number of specifics that you desire to be advertised out of rack in the case above, so it would look something like this.
hosts (/32) -> rack-sw -> (/26 + N * /32_external) -> aggregation-sw -> (/17)
On the note about label switching versus ip forwarding, I would not think of it as either/or but rather both could be used since you need to eventually serve pages to users.
hosts (/32) -> rack-sw -> (/26) -> aggregation-sw -> (/17)
If you need some specifics to be advertised, scale would be dependent on the number of specifics that you desire to be advertised out of rack in the case above, so it would look something like this.
hosts (/32) -> rack-sw -> (/26 + N * /32_external) -> aggregation-sw -> (/17)
On the note about label switching versus ip forwarding, I would not think of it as either/or but rather both could be used since you need to eventually serve pages to users.
Yeah, i was specifically thinking of the prefixes jn the tor and agg layer. I read it as having all the vips in the rack as next hop /32s. Add a bunch of intra rack ecmp groups and its not insignificant. Im not sure i buy the feasability of /27 per rack or a simple /17 supernet at the agg layer long term. Rack address counts will not be uniform, id expect everything from /29s to /23s depending on host types. Assuming uniform allocation at the pod/dc level also seems like a stretch. Renumbering and super/subnetting is going to fragment the ip locality over time. Ecmping through the core eats yet more prefixes, and (Id guess) forces hot ones to be deagg'd.
All that said, you do get a couple thousand prefixes on your $10,000 trident box. And im obviously not an NE. So maybe it would work inside a small cluster.
All that said, you do get a couple thousand prefixes on your $10,000 trident box. And im obviously not an NE. So maybe it would work inside a small cluster.
BGP was designed for AS-level routing, but that doesn't mean it can't be adapted to work at a smaller scale. iBGP is for in-network routing (as compliment to eBGP), although I couldn't say whether they are using it directly or a variant.
When you talk about 'slower than a spanning tree', that's presumably for route stabilization. But that's also largely irrelevant - routes would change infrequently enough that extremely fast stabilization isn't especially important (and on a LAN, BGB would still stabilize quite quickly). In comparison, BGP lets you add all sorts of routes to your internal network, and spread load among them without hotspotting. It can handle routes of varying sizes, loops and parallel paths, etc. Spanning trees on the other hand will pick one path from point A to point B, which won't necessarily be the best general solution for a network topology. Say you want to add a 'skip list' of rack-to-rack connections, double up others, and use two parallel DC-level tree networks because you're hitting backbone throughput limits with just one. BGP will be much better able to capitalize on this crazy mesh of connections then simpler protocols like building spanning trees.
When you talk about 'slower than a spanning tree', that's presumably for route stabilization. But that's also largely irrelevant - routes would change infrequently enough that extremely fast stabilization isn't especially important (and on a LAN, BGB would still stabilize quite quickly). In comparison, BGP lets you add all sorts of routes to your internal network, and spread load among them without hotspotting. It can handle routes of varying sizes, loops and parallel paths, etc. Spanning trees on the other hand will pick one path from point A to point B, which won't necessarily be the best general solution for a network topology. Say you want to add a 'skip list' of rack-to-rack connections, double up others, and use two parallel DC-level tree networks because you're hitting backbone throughput limits with just one. BGP will be much better able to capitalize on this crazy mesh of connections then simpler protocols like building spanning trees.
"Proceed and Be Bold"
And the anti-correlation between those who chestbeat stuff like this and those who actually do it grows stronger...
And the anti-correlation between those who chestbeat stuff like this and those who actually do it grows stronger...
We want to shoot for all of the things we want in open switches, but you can’t boil the ocean. We want hardware disaggregated from software, and we want to deploy that hardware in production.
I can't decide if this sounds a bit defeatist or is just pragmatic - isn't the goal lofty enough to require the ambition to boil the ocean?
I can't decide if this sounds a bit defeatist or is just pragmatic - isn't the goal lofty enough to require the ambition to boil the ocean?
Not really - look at Nicira, which was bought by VMware for a cool $1B. There's a lot of innovative work happening in the network virtualization space at the moment.
Sure, we are trying through global warming :)
On a serious note, take a look at cumulus networks[1] who claim to solve the hw/sw disaggregation problem. They have a linux OS distro which can run on h/w of multiple vendors ( not the popular ones like cisco/jnpr/hp/brcd etc.. since those are closed platforms).
[1] http://cumulusnetworks.com/product/overview/
On a serious note, take a look at cumulus networks[1] who claim to solve the hw/sw disaggregation problem. They have a linux OS distro which can run on h/w of multiple vendors ( not the popular ones like cisco/jnpr/hp/brcd etc.. since those are closed platforms).
[1] http://cumulusnetworks.com/product/overview/
I so wish that pricing was available for any of their supported hardware :/
"It starts up, it has a BIOS environment to do its diagnostics and testing, and then it will look for an executable and go find an operating system."
All switches on the internet booting from the network? That's the wet dream of every hacker, including the NSA and their colleagues in other countries. The cherry on the cake would be if these things could get a BIOS update without human intervention. I hope they get the security right.
All switches on the internet booting from the network? That's the wet dream of every hacker, including the NSA and their colleagues in other countries. The cherry on the cake would be if these things could get a BIOS update without human intervention. I hope they get the security right.
All of Facebook's servers already operate this way. The network is more sensitive but probably not by much.
He's right - this is definitely where the network needs to go. When we can manipulate network topology as easily as we can provision cloud-based servers today, we'll unimaginably more flexible than we are today.
Virtual topology. Someone is still running cables at the end of the day. And fat switch fabrics have some scary cabling indeed.
I was able to see a preview of the latest OpenStack a friend of mine has running on his cluster and was quite impressed where they are at currently. Although I would much rather see solid API and have communities (or startups) build products on top barebones installations like this.
Can a more knowledgeable person please explain what he means by that? I was under the impression that BGP was only for inter-ISP (or "Autonomous Network") connections, and that its slower than spanning tree.