High availability load balancing with HAProxy and CARP on FreeBSD(techbar.me)
techbar.me
High availability load balancing with HAProxy and CARP on FreeBSD
http://www.techbar.me/2013/01/high-availability-load-balancing-with-haproxy-on-freebsd/
4 comments
Also if you're setting up in a virtual environment keep in mind that it requires promiscuous mode enabled.
This stopped the show for me once when setting up two OpenBSD load balancers in a shared virtual environment. I was told that to enable promiscuous mode on a single port group they would also have to enable it on the physical ESX host adapter for each ESX host since the pair was separated on different physical hosts.
If that is true then I would never enable it. However networking isn't my strong side so I can't verify this.
This stopped the show for me once when setting up two OpenBSD load balancers in a shared virtual environment. I was told that to enable promiscuous mode on a single port group they would also have to enable it on the physical ESX host adapter for each ESX host since the pair was separated on different physical hosts.
If that is true then I would never enable it. However networking isn't my strong side so I can't verify this.
Well generally you're going to be putting multicast traffic on a default subnet of something like 224.0.0.0/4.
It's pretty easy to filter that out using iptables as well. I know for keepalived, you're generally only sending out notifications every second or two. Anything higher than that and yeah, you could end up generating some crazy traffic.
Here's a typical tcpdump of an advertisement for the curious: 172.20.1.xx > 224.0.0.18: AH(xxxxx): VRRPv2, Advertisement, vrid 51, prio 104, authtype ah, intvl 1s, length 20, addrs: 172.20.1.xx
It's pretty easy to filter that out using iptables as well. I know for keepalived, you're generally only sending out notifications every second or two. Anything higher than that and yeah, you could end up generating some crazy traffic.
Here's a typical tcpdump of an advertisement for the curious: 172.20.1.xx > 224.0.0.18: AH(xxxxx): VRRPv2, Advertisement, vrid 51, prio 104, authtype ah, intvl 1s, length 20, addrs: 172.20.1.xx
Thanks, you are right about multicast traffic which I totally forgot to mention, I will have that in mind...
Carp/ucarp are pretty fun. Additionally keepalived in linux implements vrrp and some additional nice features such as scripting what to do on failover of resources.
An addendum to this guide might be to add in connection tracking across the master/slave nodes. In *BSD this is implemented with pfsync. In linux it would be iptables connection tracking.
More on pfsync: http://www.openbsd.org/faq/pf/carp.html#pfsyncop Linux conntrackd: http://conntrack-tools.netfilter.org/
Edit: Oh, and everyone always forgets about using LVS for load-balancing and failover. There's endless documentation on the web about that, and it's not a proxy service like haproxy (which is both good and bad).
An addendum to this guide might be to add in connection tracking across the master/slave nodes. In *BSD this is implemented with pfsync. In linux it would be iptables connection tracking.
More on pfsync: http://www.openbsd.org/faq/pf/carp.html#pfsyncop Linux conntrackd: http://conntrack-tools.netfilter.org/
Edit: Oh, and everyone always forgets about using LVS for load-balancing and failover. There's endless documentation on the web about that, and it's not a proxy service like haproxy (which is both good and bad).
You are right, connection tracking is must have in production environments, I will try both pfsync and iptables. Thanks
Well pfsync is only for freebsd pf. If you use linux instead then you'll be using iptables + conntrackd.
They're both powerful systems with a different way of thinking about the idea of packet filtering and mangling!
They're both powerful systems with a different way of thinking about the idea of packet filtering and mangling!
(pf is actually taken from openbsd and ported to freebsd every few release cycles, so pf/pfsync/carp is an openbsd-ism made available to others by the generosity of using the BSD license.)
The PF that is currently in 9.1 is a few releases behind from OpenBSD and most likely will stick that way because of major changes. In FreeBSD the PF is now fully multi-core aware and thus granularly locked providing a lot of extra performance, this does mean that importing the latest changes from OBSD into FBSD is going to be more difficult!
Interesting, didn't know that! How do the multi-core abilities of PF compare to how Linux handles iptables (interrupt handling, which multi-core depends on your kernel supporting it)?
I don't know. I don't use iptables. The last time I deployed Linux was for an Android Continuous Integration server since the SDK won't run on the BSD's.
Is this load balancing or just a redundant online backup? The way I read this was that all traffic goes to one server until it fails and then it flows to the backup. If this is the case and you got a huge influx of traffic it seems like it would crash server 1 and then 2. Or am I misunderstanding it?
You're slightly misunderstanding things.
The article describes two things:
1. Having a virtual IP which can move around between two physical hosts, such that it is "always available". (It will clearly go away if both hosts crash).
2. Using HAProxy to route incoming requests, from the single virtual IP, to both back-end webservers.
This means in the expected & typical scenario where both hosts are online both webservers will handle half the load. When one host fails the other will handle all traffic.
The article describes two things:
1. Having a virtual IP which can move around between two physical hosts, such that it is "always available". (It will clearly go away if both hosts crash).
2. Using HAProxy to route incoming requests, from the single virtual IP, to both back-end webservers.
This means in the expected & typical scenario where both hosts are online both webservers will handle half the load. When one host fails the other will handle all traffic.
Ok, if both load balancers go away it will crash. When master or backup is up, both webservers in background can take all load (100%), because just one loadbalancer will work in one time.
This is both load balancing and a redundant load balancer.
Traffic goes to all the backend web-servers under this scenario, through a single load-balancer device/server.
If that device goes down the secondary takes over.
While if you have millions of requests/sec this scenario wouldn't be quite enough for you, using haproxy with carp/ucarp/keepalived will get you at least 100k connections or more. If you need millions, then you have other problems to worry about as well :).
Traffic goes to all the backend web-servers under this scenario, through a single load-balancer device/server.
If that device goes down the secondary takes over.
While if you have millions of requests/sec this scenario wouldn't be quite enough for you, using haproxy with carp/ucarp/keepalived will get you at least 100k connections or more. If you need millions, then you have other problems to worry about as well :).
Basically this will not increase performance, just availability as the main problem with load balancing is that it becomes a SPOF, if it goes down everything is down no matter how powerful web servers you have in background. What will happen if one of those gets huge amount of traffic... I think that it will not go to the backup as CARP should work. But yes it would be nice to test that.
Does anyone know how CARP on FreeBSD compares to keepalived on Linux?
CARP compares well to UCARP on Linux, as I recently documented here:
* http://www.debian-administration.org/article/678/Virtual_IP_...
* http://www.debian-administration.org/article/678/Virtual_IP_...
Thanks, I didn't know about UCARP
I didn't try keepalived on Linux (I tried it on FreeBSD) so I'm not right person to say is it better or not. Only I can say is that CARP performed really well from my perspective. I'm also interested in comparison of those two...
Keepalived implements vrrp, which seems to have patent issues according to Wikipedia https://en.wikipedia.org/wiki/Common_Address_Redundancy_Prot...
We did have some network issues after the new topology went live that we unexpectedly tracked down to the LB pair, for one simple reason: CARP generates a lot of multicast traffic. Depending on how your hosts and network are configured, this can easily get routed out into a fairly large portion of your local network, and use a lot of bandwidth/router capacity with no benefit.
So, if you're setting up a CARP pair/cluster of your own, pay close attention to your multicast setup. Ideally, put the CARP multicast traffic on a dedicated subnet and watch your router and switch stats to make sure you aren't flooding the rest of your network with multicast spam.