Hi pm90, I work on the facebook network engineering team, I can explain at a high-level some of the reasons we made this change.
You're correct BGP was originally intended for inter-AS routing, however, those AS's don't necessarily have to be different ISPs. They could be different divisions in a company, or different racks in a data-center. They can be any grouping where you might want different policies applied. You can also use BGP between devices in the same AS where a common policy makes sense. There's no technical limitation stopping you from using BGP inside the data-center, so it comes down to pros and cons of the various options out there.
Our main goals were scalability, reliability and simplicity. When it comes to scalability BGP is essentially the best, this is because, and why it's used between AS's where the routing tables can grow quite large. When it comes to reliability, we previously had a layer2 architecture, but there are a lot of challenges with that at scale, some examples:
* Split brain: In a layer2 world, you still typically have a layer3 protocol running between your upper layers. We were using BGP and VRRP at the time, and issues in one wouldn't always be communicated to the other, in some cases that's not possible. So VRRP may think everything is fine, while BGP is down, the result is black-holed trafic.
* Scale: Running an all layer2 network requires devices that have large enough CAM tables to support all connected devices, many vendors newer full line rate cards were coming out with smaller CAM tables, as such layer2 simply wasn't an option in some cases. Additionally, many of the other protocols that were options are computationally expensive to deploy at a large size, you either end up slowing them down, or deploying relatively complex designs.
One of the many significant benefits BGP brought us came in the form of ECMP. In a layer2/VRRP world, balancing traffic is hacky with two destinations, if you want to do more, which we did, it becomes increasingly complex. With BGP it's built in, we can consistently and easily balance the load to 8, 16, 32 or more destinations depending
on the hardware and software in use.
BGP is also readily available in the software from the major network vendors, as well as within OpenSource solutions for the servers. This means we can use a common protocol everywhere. Which in turns makes sourcing and deploying new equipment much easier and simplifies the configs, policies and tooling needed to support the deployment.
There are many other reasons that layer2 wasn't a good choice for us, and that layer3 makes a lot of sense. I'd be happy to discuss more of these as well.
You're correct BGP was originally intended for inter-AS routing, however, those AS's don't necessarily have to be different ISPs. They could be different divisions in a company, or different racks in a data-center. They can be any grouping where you might want different policies applied. You can also use BGP between devices in the same AS where a common policy makes sense. There's no technical limitation stopping you from using BGP inside the data-center, so it comes down to pros and cons of the various options out there.
Our main goals were scalability, reliability and simplicity. When it comes to scalability BGP is essentially the best, this is because, and why it's used between AS's where the routing tables can grow quite large. When it comes to reliability, we previously had a layer2 architecture, but there are a lot of challenges with that at scale, some examples:
* Split brain: In a layer2 world, you still typically have a layer3 protocol running between your upper layers. We were using BGP and VRRP at the time, and issues in one wouldn't always be communicated to the other, in some cases that's not possible. So VRRP may think everything is fine, while BGP is down, the result is black-holed trafic.
* Scale: Running an all layer2 network requires devices that have large enough CAM tables to support all connected devices, many vendors newer full line rate cards were coming out with smaller CAM tables, as such layer2 simply wasn't an option in some cases. Additionally, many of the other protocols that were options are computationally expensive to deploy at a large size, you either end up slowing them down, or deploying relatively complex designs.
One of the many significant benefits BGP brought us came in the form of ECMP. In a layer2/VRRP world, balancing traffic is hacky with two destinations, if you want to do more, which we did, it becomes increasingly complex. With BGP it's built in, we can consistently and easily balance the load to 8, 16, 32 or more destinations depending on the hardware and software in use.
BGP is also readily available in the software from the major network vendors, as well as within OpenSource solutions for the servers. This means we can use a common protocol everywhere. Which in turns makes sourcing and deploying new equipment much easier and simplifies the configs, policies and tooling needed to support the deployment.
There are many other reasons that layer2 wasn't a good choice for us, and that layer3 makes a lot of sense. I'd be happy to discuss more of these as well.
Hopefully this helps to answer your question.