Zero Downtime Frontend Deploys with Vulcand on CoreOS(coreos.com)
coreos.com
Zero Downtime Frontend Deploys with Vulcand on CoreOS
http://coreos.com/blog/zero-downtime-frontend-deploys-vulcand/
23 comments
The architecture is sound. But half-assing a loadbalancer impl (vulcand) instead of embedding HAProxy is a terrible idea.
+1
confd + haproxy would be a better choice.
I have plans do roll the cocktail out, but running the HA prox y on each machine, much like the airbnb smartstack: http://nerds.airbnb.com/smartstack-service-discovery-cloud/
confd + haproxy would be a better choice.
I have plans do roll the cocktail out, but running the HA prox y on each machine, much like the airbnb smartstack: http://nerds.airbnb.com/smartstack-service-discovery-cloud/
So it's half assed if it's not HAProxy? How so?
HAProxy is 14 years old, under active development by very competent people, used in thousands of production deployments and many hardware appliances.
Pretty much every large site you visit every day has HAProxy in the stack. It's one of the cornerstones of OSS infrastructure software.
Are you seriously asking how a 2 month old hack (vulcand) is half-assed in comparison?
Pretty much every large site you visit every day has HAProxy in the stack. It's one of the cornerstones of OSS infrastructure software.
Are you seriously asking how a 2 month old hack (vulcand) is half-assed in comparison?
If you're going to call something half-assed because it's not HAProxy then I for one would consider it more productive if this was framed in terms of metrics instead of something like "half-assed." Does HAProxy have a suite of metrics that can serve as benchmarks for all the projects out there that would dare to be less half-assed?
Why didn't you ask the question the other way around -- does vulcand have anything to show that it is better in any way than the current best-practice load balancer?
Reading the docs, the only thing that stands out is that it's tightly coupled with etcd, which seems like a really bad idea architecturally.
Reading the docs, the only thing that stands out is that it's tightly coupled with etcd, which seems like a really bad idea architecturally.
I'm wondering why do you think that tightly coupling with Etcd is a bad idea?
Etcd provides advanced features for configuration and service discovery - serialized updates, high availability and ttls for keys that's why it was a reasonable choice for me to use it and not file system that does not provide any of the above features out of the box.
One of the main reasons I wrote Vulcand is for experimentation - what would be the next steps for proxy and load balancing out there?
One of the ideas - having a proxy that will embrace the dynamic nature of the Docker enabled backends an provide full support for rapidly changing dockerized landscape.
Another idea is to try out some ideas in error recovery/failure detection algos, like preventing cascading failures:
http://mailgun.github.io/vulcan-docs/#weighted-round-robin
I don't pretend that Vulcand is somewhat robust and everyone should immediately drop their HAProxy/Nginx solutions and start using Vulcand. I hoped that would be clear from the README and is highlighted in Rob's post as well.
In fact, I'm using Vulcand as well as Nginx and HAproxy, I'm just choosing carefully when should I use one load balancer vs another.
Etcd provides advanced features for configuration and service discovery - serialized updates, high availability and ttls for keys that's why it was a reasonable choice for me to use it and not file system that does not provide any of the above features out of the box.
One of the main reasons I wrote Vulcand is for experimentation - what would be the next steps for proxy and load balancing out there?
One of the ideas - having a proxy that will embrace the dynamic nature of the Docker enabled backends an provide full support for rapidly changing dockerized landscape.
Another idea is to try out some ideas in error recovery/failure detection algos, like preventing cascading failures:
http://mailgun.github.io/vulcan-docs/#weighted-round-robin
I don't pretend that Vulcand is somewhat robust and everyone should immediately drop their HAProxy/Nginx solutions and start using Vulcand. I hoped that would be clear from the README and is highlighted in Rob's post as well.
In fact, I'm using Vulcand as well as Nginx and HAproxy, I'm just choosing carefully when should I use one load balancer vs another.
First, let me say up front that I value your verve and appreciate your desire to experiment and learn and push frontiers. Bravo on all that.
The problem with tight coupling -- especially across concerns and software layers, as with 'etcd -> load balancer' -- is that it's an immediate withdrawal from the great Bank of Technical Debt.
Some time in the future, one of the following things will happen:
1. Etcd will want to perform one unit of positive technical change, and either that behavior will break your code, or you will be forced to fork versions, or etcd will be forced not to make that change by the popularity of your code.
2. You will want to perform one unit of positive technical change, but etcd will, for whatever political, architectural, or technical reason not wish to support this change. You will either abandon your idea for lack of upstream technical support or be forced to fork etcd.
The neglect of these incredibly common occurrences is itself now incredibly common. It underpins, e.g., systemd development, which in turn appears to be informing coreos and docker development, and since those are the new-shiny-toys, a lot of people seem to think that's just the way we should all think now, that the 'unix philosophy' is ripe for disruption(tm).
Problem is, invariably, coupling turns out to be the most important artifact in all of computer science. It's worse than naming things, it's worse than cache invalidation, it's worse than off-by-one errors. And the worst kind of coupling is the kind of coupling that crosses stack boundaries for no obviously good reason, because the amount and kind of reasoning that has to happen at an operational level when that stops working in production is both severe and very difficult to resolve.
So again: applaud the desire to learn, and experiment. The coupling: please, please don't. The marketing of a test solution as something to consider using: you don't need that to build your portfolio, just put the project up and say it's a test thing. And preventing cascading failures with weighted round robin, etc: http://cbonte.github.io/haproxy-dconv/configuration-1.5.html...
The problem with tight coupling -- especially across concerns and software layers, as with 'etcd -> load balancer' -- is that it's an immediate withdrawal from the great Bank of Technical Debt.
Some time in the future, one of the following things will happen:
1. Etcd will want to perform one unit of positive technical change, and either that behavior will break your code, or you will be forced to fork versions, or etcd will be forced not to make that change by the popularity of your code.
2. You will want to perform one unit of positive technical change, but etcd will, for whatever political, architectural, or technical reason not wish to support this change. You will either abandon your idea for lack of upstream technical support or be forced to fork etcd.
The neglect of these incredibly common occurrences is itself now incredibly common. It underpins, e.g., systemd development, which in turn appears to be informing coreos and docker development, and since those are the new-shiny-toys, a lot of people seem to think that's just the way we should all think now, that the 'unix philosophy' is ripe for disruption(tm).
Problem is, invariably, coupling turns out to be the most important artifact in all of computer science. It's worse than naming things, it's worse than cache invalidation, it's worse than off-by-one errors. And the worst kind of coupling is the kind of coupling that crosses stack boundaries for no obviously good reason, because the amount and kind of reasoning that has to happen at an operational level when that stops working in production is both severe and very difficult to resolve.
So again: applaud the desire to learn, and experiment. The coupling: please, please don't. The marketing of a test solution as something to consider using: you don't need that to build your portfolio, just put the project up and say it's a test thing. And preventing cascading failures with weighted round robin, etc: http://cbonte.github.io/haproxy-dconv/configuration-1.5.html...
Does HAProxy have a suite of metrics that can serve as benchmarks for all the projects out there that would dare to be less half-assed?
The overwhelming metric is "how soon can I use this in my production deployment without worrying about it".
For HAProxy this becomes true (as far as its role is concerned) as soon as the integration points with CoreOS are stable.
For vulcand it's unknown when and if it may become true.
I'm not sure what other metrics you ask for that are not provided on the HAProxy homepage?
The overwhelming metric is "how soon can I use this in my production deployment without worrying about it".
For HAProxy this becomes true (as far as its role is concerned) as soon as the integration points with CoreOS are stable.
For vulcand it's unknown when and if it may become true.
I'm not sure what other metrics you ask for that are not provided on the HAProxy homepage?
We have something very similar at Balanced called Rump (routing umpire). It's written in python has a pretty cool rule based engine and a swappable backend.
It currently uses ZK and Chef, but it would be pretty interesting to use something like Etcd/Consul since no one wants to maintain a ZK cluster.
If there are individuals who want to hack on it, shoot me a msg on twitter and I'll see what I can do to speed up the process to open source it.
It currently uses ZK and Chef, but it would be pretty interesting to use something like Etcd/Consul since no one wants to maintain a ZK cluster.
If there are individuals who want to hack on it, shoot me a msg on twitter and I'll see what I can do to speed up the process to open source it.
What great timing! I was just about to roll my own using etcd, haproxy and a bunch of perl duct tape.
Very excited to take a good look at vulcand.
Very excited to take a good look at vulcand.
If you're still interested in doing this for any reason, check out confd [0] for populating templates based on etcd values.
[0] https://github.com/kelseyhightower/confd
[0] https://github.com/kelseyhightower/confd
Yes, confd was on my radar and looks promising. In my case, we've already got a wad of perl duct-tape out there and adding etcd was a baby step. Using confd would have been a later step.
Vulcand will likely allow me to replace the whole shebang in one step.
Vulcand will likely allow me to replace the whole shebang in one step.
it doesn't look like vulcan supports HTTPS, so it's a no-go for anything serious. Here's a post on my confd/nginx/docker setup for zero-downtime using etcd. http://brianketelsen.com/2014/02/25/using-nginx-confd-and-do...
I actually learned about confd via your post. Thanks for that.
In our case, HTTPS isn't an issue but there are good arguments for using a tried-and-true proxy. I see Vulcan as a quick way to test out docker containers for our architecture.
In the end I'll probably use confd to rewrite a VCL file for Varnish. It's just a matter of deciding how many containers and what level of sidekicking/ambassadoring I'm willing to put up with to "properly" organize things...
In our case, HTTPS isn't an issue but there are good arguments for using a tried-and-true proxy. I see Vulcan as a quick way to test out docker containers for our architecture.
In the end I'll probably use confd to rewrite a VCL file for Varnish. It's just a matter of deciding how many containers and what level of sidekicking/ambassadoring I'm willing to put up with to "properly" organize things...
Yep, until this won't be closed:
https://github.com/mailgun/vulcan/issues/31
Vulcand should not be used as for anything frontend-related
https://github.com/mailgun/vulcan/issues/31
Vulcand should not be used as for anything frontend-related
I'll ask about the reasoning instead of attacking the decision (since that didn't get a response).
Why not embed HAProxy?
Why not embed HAProxy?
etcd, fleet, deis, and now this are getting me more and more tempted to move from Velociraptor to the Docker ecosystem.
If Deis could run my uptests[0] before Vulcan gets called to re-route things, that'd be awesome.
https://bitbucket.org/yougov/velociraptor/src/2bcef54253a9e9...
If Deis could run my uptests[0] before Vulcan gets called to re-route things, that'd be awesome.
https://bitbucket.org/yougov/velociraptor/src/2bcef54253a9e9...
Come find us in #deis on Freenode and we'll see if we can help.