Getting the most out of HAProxy(twilio.com)

https://www.twilio.com/engineering/2013/10/16/haproxy

16 comments

Want to know what else is cool about haproxy? The stats unix socket. With it, you can query the existing configuration at whim, and with haproxy 1.4 or greater, you can configure haproxy on the fly.

    show stat -1 4\n
    disable server xyz\n

What's not so cool? Disabling a server does nothing to kill off existing connections to that server, and idle timeouts can cause problems if your processes are slow to respond.

With care, though, it's a great component of a HA solution, in addition to a being a great load balancer.

nasalgoat13년 전

If you're running multiple haproxy processes, the unix socket commands only work on the process you happen to randomly get when you use the socket.

I've asked about plans to fix that and it's not on the current roadmap for 1.5.

I need multiple processes to allow SSL offloading.

falcolas13년 전

Perhaps start each up with a separate configuration file for each instance? Or run as a cluster, so changes to one are propagated to all (untested)?

    haproxy -f /path/to/alternate/cfg

nl13년 전

As we convert more and more Twilio tools to a service-oriented architecture, these logs have been invaluable at debugging failures

I'd love to know more about how Twilio does SOA (and no, the linked document doesn't expand on it).

Do they use an ESB or do they rely on individual services connecting directly to each other?

I'm 100% convinced of the ideas behind a service-oriented architecture. I'm less convinced about the need for an ESB, but I'm happy to be talked around.

Experiences/Opinions/War Stories eagerly sought.

kevinburke13년 전

Internally each FE runs HAProxy to round robin between all of the shared backends, no ESB involved. Hoping to write another blog post about this soon.

Agree that the link is kind of not-good, this might help a little... http://www.slideshare.net/twilio/highavailability-infrastruc...

I wish I could say more about our infrastructure and experience with SOA, but I'm a little limited in what I can share. We are hiring though :)

nl13년 전

Thanks for the response & link.

What's a "FE"?

llakey13년 전

I suspect it's short for: Front End

gopalv13년 전

More fun HAProxy bits include pinning a user persistently to one machine (which fails over when that machine fails a health-check)

Particularly useful when you've got a web+mc combo in one machine which can get the most out of such a setup - http://notmysock.org/blog/hacks/haproxy-user-repinning

petercooper13년 전

I don't do anything heavy enough to need load balancing, but I've always found HAProxy to be ideal for situations where I'm limited by IPs but I want to run a variety of HTTP oriented services on the same machine.

It's so easy to filter by hostname and send requests off to different internal daemons and HAProxy is solid as a rock unlike other mechanisms I've tried.

jtokoph13년 전

Another cool HAProxy option is setting up backup/failover backends. So if all of your real backends die or go offline, haproxy can serve a friendly maintenance/downtime page from the failover backend.

zerop13년 전

where haproxy scores above ngingx..I am curious to know, as we use nginx....

zh13년 전

NginX is very good WEB SERVER - so it shines in serving content - web pages, caching images etc. But in proxy mode, the load-balancing between backends is pretty basic. HAProxy is born to be very good LOAD BALANCER - you can have different balancing algorithms, not only round-robin. HAProxy have also TCP mode, so you can balance your MySQL backends, for example. Also for me, personally, HAProxy stats rock.

vidarh13년 전

Regarding your point about MySQL:

We used haproxy to do close to zero downtime migration of our postgres servers (and other stuff) from one data centre to another recently when we moved out of our old data centre: Set up a slave in the new dc, set up haproxy with the slave as backup, point all the clients to haproxy. Shut down the master, and as soon as the master stopped responding, haproxy would shove clients to the slave in the new data centre instead. Then touch the trigger file to make the slave recover and switch to master mode.

It meant we were in read-only mode because requests where going to the slaves for about 2-3 seconds (and unlike MySQL Postgres does not allow writes to a slave until it's been promoted) before the slave started handling updates - other than that everything just chugged along.

ZoFreX13년 전

Thanks for the explanation, it's very enlightening to someone (me) who's never used haproxy. For front-end load balancing, I've been using Varnish. Is HAProxy also better at http load balancing than Varnish as well as nginx, do you know?

unkoman13년 전

I use both, as varnish does caching well. Two haproxy servers where one is a fail over. varnish on both.

dantheta13년 전

I was also pleased to find that in version 1.4 a MySQL login health check was added "option mysql-check".