Etcd 0.3.0 – Improved Cluster Discovery, API Enhancements and Windows Support(coreos.com)
coreos.com
Etcd 0.3.0 – Improved Cluster Discovery, API Enhancements and Windows Support
https://coreos.com/blog/etcd-0.3.0-released/
49 comments
Keep in mind that Google's Chubby paper describes a nested series of clusters. You basically have one global cluster that stores the location of the cluster running in each DC. If you're on the west coast and need to read info from the east coast, you can travel up the chain and back down again. At some point you have a single source of truth. This method is basically adding another feature to the global cluster, offering a bootstrapping mechanism for other regional clusters.
This is why top-down, single-source of truth is mostly the best way for core infrastructure. No pun intended. It's important for some parts to not be infinitely reconfigurable or dynamic because it would create chaos and service dependency deadlock/DoS. Most people not running bare metal don't know the lessons of how and why things underneath are they way they are and the sensible limits what's possible.
It's also more secure because if it's possible to dictate all of the service details, it's much easier to run a lean and locked-down infrastructure.
CoreOS by itself looks great. More JEOS distros need to happen. Trying to bundle more unproven things just looks marketing led the product.
It's also more secure because if it's possible to dictate all of the service details, it's much easier to run a lean and locked-down infrastructure.
CoreOS by itself looks great. More JEOS distros need to happen. Trying to bundle more unproven things just looks marketing led the product.
This is why top-down, single-source of truth is mostly the best way for core infrastructure.
What happens when your single source of truth goes down?
etcd attempts to provide a source of truth by using the Raft[1] consensus algorithm to determine that "truth" in an environment when relying on merely one place is too risky.
Th Raft and Paxos[2] (as used in Zookeeper & Google Chubby) algorithms give guarantees for consensus in an environment with unreliable hardware. That is better than a single source of truth where you get nothing when that source disappears.
[1] http://raftconsensus.github.io/
[2] http://en.wikipedia.org/wiki/Paxos_(computer_science)
What happens when your single source of truth goes down?
etcd attempts to provide a source of truth by using the Raft[1] consensus algorithm to determine that "truth" in an environment when relying on merely one place is too risky.
Th Raft and Paxos[2] (as used in Zookeeper & Google Chubby) algorithms give guarantees for consensus in an environment with unreliable hardware. That is better than a single source of truth where you get nothing when that source disappears.
[1] http://raftconsensus.github.io/
[2] http://en.wikipedia.org/wiki/Paxos_(computer_science)
> This is why top-down, single-source of truth is mostly the best way for core infrastructure.
"Top-down, single-source of truth" needs to deal with failover for resilience in any serious setup. The entire point of consensus algorithms like Raft (and Zab, since you've mentioned ZooKeeper) is that they provide a method of providing a consistent source of truth that is resilient without the management overhead.
It's explicitly because top-down solutions are a massive hassle to keep consistent and available in the face of stalled replication (OpenLDAP.... DNS based solutions...) or a failed master.
> It's important for some parts to not be infinitely reconfigurable or dynamic because it would create chaos and service dependency deadlock/DoS.
That is entirely irrelevant to the algorithm used to replicate the configuration information.
> It's also more secure because if it's possible to dictate all of the service details, it's much easier to run a lean and locked-down infrastructure.
Why exactly do you believe that it is not "possible to dictate all of the service details" while using Etcd? To me it's an entirely orthogonal issue.
"Top-down, single-source of truth" needs to deal with failover for resilience in any serious setup. The entire point of consensus algorithms like Raft (and Zab, since you've mentioned ZooKeeper) is that they provide a method of providing a consistent source of truth that is resilient without the management overhead.
It's explicitly because top-down solutions are a massive hassle to keep consistent and available in the face of stalled replication (OpenLDAP.... DNS based solutions...) or a failed master.
> It's important for some parts to not be infinitely reconfigurable or dynamic because it would create chaos and service dependency deadlock/DoS.
That is entirely irrelevant to the algorithm used to replicate the configuration information.
> It's also more secure because if it's possible to dictate all of the service details, it's much easier to run a lean and locked-down infrastructure.
Why exactly do you believe that it is not "possible to dictate all of the service details" while using Etcd? To me it's an entirely orthogonal issue.
Quoting the post:
In previous releases, a human had to pick a leader, pass a list of -peers to all of the followers and maintain that list going forward.
Presumably that method is still there?
In previous releases, a human had to pick a leader, pass a list of -peers to all of the followers and maintain that list going forward.
Presumably that method is still there?
Yep, you can still hardcode a list of peers via flag or config file. I think you can even use both a peer list and a discovery token at the same time.
"Turtles all the way down" made my day. Brought me back to my highschool days of reading all of Hawking's books.
Discovery looks interesting. Can it be used for a client to discover a cluster?
I've been digging into Docker link containers a bit.
I'm not entirely comfortable with how they work,and I'm not really sure why. The only thing I can put my finger on is that I feel like discovery is a separate concern to deployment. But at the same time they are so closely linked o can understand why Docker needs to tackle it.
Is there a way etcd can work better with Docker links? Maybe it could automatically read/write Docker published ENV variables or something? Though I don't think that will quite work across physical machines without some additional work.
I've been digging into Docker link containers a bit.
I'm not entirely comfortable with how they work,and I'm not really sure why. The only thing I can put my finger on is that I feel like discovery is a separate concern to deployment. But at the same time they are so closely linked o can understand why Docker needs to tackle it.
Is there a way etcd can work better with Docker links? Maybe it could automatically read/write Docker published ENV variables or something? Though I don't think that will quite work across physical machines without some additional work.
> Discovery looks interesting. Can it be used for a client to discover a cluster?
It certainly could be extended for that. We haven't done anything with discovery to help with that process yet though.
> Is there a way etcd can work better with Docker links?
There is a lot of room to explore here. Including:
- Automatically registering exposed ports from running containers into etcd for service discovery.
- Using information in etcd to manage a smart proxy that would proxy requests. This would be the "Link via an Ambassador" but etcd would help coordinate.
Both of these would be good things to build and see how they work and feel in practice.
There are also some interesting alternatives to doing the links thing such as using some network namespace tricks to avoid passing metadata around via environment variables and simply expose services directly into a container: https://coreos.com/blog/Jumpers-and-the-software-defined-loc...
It certainly could be extended for that. We haven't done anything with discovery to help with that process yet though.
> Is there a way etcd can work better with Docker links?
There is a lot of room to explore here. Including:
- Automatically registering exposed ports from running containers into etcd for service discovery.
- Using information in etcd to manage a smart proxy that would proxy requests. This would be the "Link via an Ambassador" but etcd would help coordinate.
Both of these would be good things to build and see how they work and feel in practice.
There are also some interesting alternatives to doing the links thing such as using some network namespace tricks to avoid passing metadata around via environment variables and simply expose services directly into a container: https://coreos.com/blog/Jumpers-and-the-software-defined-loc...
> Discovery looks interesting. Can it be used for a client to discover a cluster?
>> It certainly could be extended for that. We haven't done anything with discovery to help with that process yet though.
So what is the current best practice for etcd discovery? Environment variable? mDNS might be nice, too (Though I guess mDNS is kinda, almost an alternative for a subset of what etcd does). http://blogs.gnome.org/danni/2014/02/02/a-libnss-plugin-for-... looks interesting too.
I read the Jumper post thing when it was HN. I need to think about that some more.
So what is the current best practice for etcd discovery? Environment variable? mDNS might be nice, too (Though I guess mDNS is kinda, almost an alternative for a subset of what etcd does). http://blogs.gnome.org/danni/2014/02/02/a-libnss-plugin-for-... looks interesting too.
I read the Jumper post thing when it was HN. I need to think about that some more.
I'm curious to know more about the garbage collection of stale peers.
- AFAIK, etcd is built on RAFT, which relies on a 'joint majority' method of transitioning cluster membership. Are there any issues forming agreement on what the new membership should be when it's unclear what nodes are still supposed to be part of the cluster? - In the land of ZooKeeper, cluster configurations are typically very very stable, so tracking membership takes very little information. Is etcd targeted to more dynamic environments where the garbage generated by entering and leaving nodes is significant?
- AFAIK, etcd is built on RAFT, which relies on a 'joint majority' method of transitioning cluster membership. Are there any issues forming agreement on what the new membership should be when it's unclear what nodes are still supposed to be part of the cluster? - In the land of ZooKeeper, cluster configurations are typically very very stable, so tracking membership takes very little information. Is etcd targeted to more dynamic environments where the garbage generated by entering and leaving nodes is significant?
etcd isn't designed for significantly dynamic environments but we certainly want to make it easy for people to add and remove machines at runtime. We have basic support for runtime cluster configuration but are building out a more complete API. The current proposal that is being built can be found over here: http://thread.gmane.org/gmane.comp.distributed.etcd/168
As you point out there is no magic garbage collection of stale peers. You have to explicitly delete nodes that you don't expect to be participating any longer.
As you point out there is no magic garbage collection of stale peers. You have to explicitly delete nodes that you don't expect to be participating any longer.
Has anyone checked out serf http://www.serfdom.io/ if yes what are the pros and cons of etcd and serf?
Especially Windows support is cool news. I am following etcd project, it's serious and can change server configuration world. Thanks Alex!
Congratz guys, this is a big improvement from 0.2.0. Keep up good work!
This is a pure marketing story pushing a bad solution.
Hiera, a simple hierarchal property distribution system using a backend of Zookeeper plus puppet or chef is far superior. Etcd is the PHP of configuration management.
Hiera, a simple hierarchal property distribution system using a backend of Zookeeper plus puppet or chef is far superior. Etcd is the PHP of configuration management.
Wow. I know neither etcd or Hiera. But please provide some arguments against etcd if you think it is bad, rather than just bashing it.
It would be interesting to know the cons and pros of both solutions.
It would be interesting to know the cons and pros of both solutions.
I'm bashing it because it oversimplifies the problem and claims to be a solution that doesn't address the irreducibility of the problem.
It's great for a host file but scp can do that.
It doesn't coordinate the transition of services to other states. Puppet does an incredible amount of work (dependency satisfaction) to get things where you want them to be. Configuration management is as much management order of operations as well as what files should contain what. There's not much separating the two because you have to usually change them between states. Chef does this to, but in a simpler way.
Cfengine was a past attempt, but is really complicated double work of what runs on a server and what runs on a managed node.
It's great for a host file but scp can do that.
It doesn't coordinate the transition of services to other states. Puppet does an incredible amount of work (dependency satisfaction) to get things where you want them to be. Configuration management is as much management order of operations as well as what files should contain what. There's not much separating the two because you have to usually change them between states. Chef does this to, but in a simpler way.
Cfengine was a past attempt, but is really complicated double work of what runs on a server and what runs on a managed node.
I'm not sure what problem you think etcd is attempting to solve, but if your solution involvs Puppet or Chef then it isn't surprising you find etcd isn't sufficient for your purposes.
etcd is a highly available configuration store. That's all. It's designed for applications to use to store configuration, so they can be started & stopped without relying on file based storage. Generally speaking these applications have to be coded to use etcd (which is completely different to Chef/Puppet, which are Ops tools). Sure, you could do something to write etcd values to a file, and then have them used by any program, but that misses the point.
Your criticisms are so completely irrelevant they aren't even wrong.
etcd is a highly available configuration store. That's all. It's designed for applications to use to store configuration, so they can be started & stopped without relying on file based storage. Generally speaking these applications have to be coded to use etcd (which is completely different to Chef/Puppet, which are Ops tools). Sure, you could do something to write etcd values to a file, and then have them used by any program, but that misses the point.
Your criticisms are so completely irrelevant they aren't even wrong.
No bashing against etcd from me, but can you point me to a reason why a regular database wouldn't work for this type of thing?
High availability, throughput and data storage seem to fit into what a database is made for.
High availability, throughput and data storage seem to fit into what a database is made for.
You'd think so, but most RDBMS's are a lot of work to set up in a way that provides the kind of consistency guarantees and availability that a configuration store like this does.
E.g. take Postgres. You can't connect to the slaves and write, so your client will need to know how to identify which server is currently the master. And if the master fails, you need to implement a leadership election method, promote the chosen slave, and resync your other slaves from the new master.
Before you know it you've reimplemented the configuration store with a different data storage mechanism. There's very little intersection between the typical RDBMS and a configuration store like this.
E.g. take Postgres. You can't connect to the slaves and write, so your client will need to know how to identify which server is currently the master. And if the master fails, you need to implement a leadership election method, promote the chosen slave, and resync your other slaves from the new master.
Before you know it you've reimplemented the configuration store with a different data storage mechanism. There's very little intersection between the typical RDBMS and a configuration store like this.
Well, even with your point there are a couple reasons why I could imagine databases making sense here.
One is that you might already have people used to deal with them in a highly available manner in your team. HA with Postgres is not exactly an area of research anymore. Most people already using Postgres in their stack know about tools like pgbouncer.
Another one is that there are other databases, like Riak, that can talk http and have those problems solved.
I am here more trying to understand the features that would make one pick etcd over an higly available database. Thanks for taking the time to explain.
One is that you might already have people used to deal with them in a highly available manner in your team. HA with Postgres is not exactly an area of research anymore. Most people already using Postgres in their stack know about tools like pgbouncer.
Another one is that there are other databases, like Riak, that can talk http and have those problems solved.
I am here more trying to understand the features that would make one pick etcd over an higly available database. Thanks for taking the time to explain.
They solve different problems.
In CAP theorem[1] terms, etcd provides Consistency and Partition-Tolerance, while databases generally provide Consistency and Availability.
That's important, because it shows the kind of problem each is trying to solve.
[1] http://en.wikipedia.org/wiki/CAP_theorem
In CAP theorem[1] terms, etcd provides Consistency and Partition-Tolerance, while databases generally provide Consistency and Availability.
That's important, because it shows the kind of problem each is trying to solve.
[1] http://en.wikipedia.org/wiki/CAP_theorem
Great, that's exactly the answer I was looking for. Now everything makes sense.
The HTTP based interface[1] is designed specifically for configuration data and is easily callable from shell scripts as well as form programs.
Something like CouchDB is roughly comparable in that sense, but that has pretty different design goals.
[1] https://github.com/coreos/etcd/blob/master/Documentation/api...
Something like CouchDB is roughly comparable in that sense, but that has pretty different design goals.
[1] https://github.com/coreos/etcd/blob/master/Documentation/api...
> It doesn't coordinate the transition of services to other states.
It's not intended to. It is way outside the scope. But you can easily coordinate transitions on top of Etcd by waiting on key changes.
> Configuration management is as much management order of operations as well as what files should contain what.
Outside of scope. Nothing stops you from using a configuration management engine like Puppet and triggering it on state changes by waiting for Etcd keys. It'd be trivial.
Or use something other than Etcd if those are your needs. But those are not my needs.
For my configuration needs there are exactly no places where I have any interest in the complexity of Puppet, but plenty of places where being able to easily set and query configuration values via a simple REST api that's trivially to access even from a bash script is nice.
You're right that scp can do that. But then you're suddenly in the job of maintaining hairy scripts to push out config changes and ensure they are in sync. So Etcd provides a solution that provides that synchronization without the complexity of far more featureful solutions. To me, that is a feature.
You seem to try to evaluate a handsaw as if it was a chainsaw and give it bad marks for being cumbersome to cut down trees with. Not all of us want to cut down trees.
It's not intended to. It is way outside the scope. But you can easily coordinate transitions on top of Etcd by waiting on key changes.
> Configuration management is as much management order of operations as well as what files should contain what.
Outside of scope. Nothing stops you from using a configuration management engine like Puppet and triggering it on state changes by waiting for Etcd keys. It'd be trivial.
Or use something other than Etcd if those are your needs. But those are not my needs.
For my configuration needs there are exactly no places where I have any interest in the complexity of Puppet, but plenty of places where being able to easily set and query configuration values via a simple REST api that's trivially to access even from a bash script is nice.
You're right that scp can do that. But then you're suddenly in the job of maintaining hairy scripts to push out config changes and ensure they are in sync. So Etcd provides a solution that provides that synchronization without the complexity of far more featureful solutions. To me, that is a feature.
You seem to try to evaluate a handsaw as if it was a chainsaw and give it bad marks for being cumbersome to cut down trees with. Not all of us want to cut down trees.
Trying to cast an argument as a strawman with a strawman is pretty weak.
You've also admitted that it's an incomplete solution. But What's worse is that people are going to buy this and quickly they'll have to scrap it when it gets out of sync, deadlocks or can't handle more than 8 boxes.
You've also admitted that it's an incomplete solution. But What's worse is that people are going to buy this and quickly they'll have to scrap it when it gets out of sync, deadlocks or can't handle more than 8 boxes.
> Trying to cast an argument as a strawman with a strawman is pretty weak.
I've not presented a strawman. I've pointed out to you that people have different needs. I don't need, nor want what Puppet provides, and Etcd does not claim to provide what Puppet provides.
I do, however want what Etcd provides.
> You've also admitted that it's an incomplete solution
In the sense that a car is an incomplete solution if you need a space shuttle.
Yet the car is still more practical if you want to drive on a road.
> But What's worse is that people are going to buy this and quickly they'll have to scrap it when it gets out of sync, deadlocks or can't handle more than 8 boxes.
What basis do you have for that claim?
I've not presented a strawman. I've pointed out to you that people have different needs. I don't need, nor want what Puppet provides, and Etcd does not claim to provide what Puppet provides.
I do, however want what Etcd provides.
> You've also admitted that it's an incomplete solution
In the sense that a car is an incomplete solution if you need a space shuttle.
Yet the car is still more practical if you want to drive on a road.
> But What's worse is that people are going to buy this and quickly they'll have to scrap it when it gets out of sync, deadlocks or can't handle more than 8 boxes.
What basis do you have for that claim?
> It doesn't coordinate the transition of services to other states.
Can you be more specific? So to co-ordinate transitions what else would it need -- multi-key transactions?
Surely you have good reason to jump in and criticize it so sharply...
Can you be more specific? So to co-ordinate transitions what else would it need -- multi-key transactions?
Surely you have good reason to jump in and criticize it so sharply...
Configuration management transactions.
Puppet has the notion of stages.
Yes because it's marketing an unproven technology as a solution when better things already exist. Someone I know from (OpsCode||Puppet) even further the recommendation of zk as basically the only proven consistent KVS. A better approach should be to do what other cloud shops have done and ship ZK appliances. Pushing new tech on customers is using them to beta test unproven tech, which is reinventing the wheel and risky.
Puppet has the notion of stages.
Yes because it's marketing an unproven technology as a solution when better things already exist. Someone I know from (OpsCode||Puppet) even further the recommendation of zk as basically the only proven consistent KVS. A better approach should be to do what other cloud shops have done and ship ZK appliances. Pushing new tech on customers is using them to beta test unproven tech, which is reinventing the wheel and risky.
[deleted]
Why is the solution bad, can you explain?
Also never heard of Hiera. I've heard of Zookeeper, Paxos & Raft protocols, also avahi, DNS, LDAP, ansible, saltstack, in other words I am not a total newcomer to system configuration management but am I the only one who hasn't heard of Hiera?
Also never heard of Hiera. I've heard of Zookeeper, Paxos & Raft protocols, also avahi, DNS, LDAP, ansible, saltstack, in other words I am not a total newcomer to system configuration management but am I the only one who hasn't heard of Hiera?
There are plenty of good things few people know about.
It's bad to promote a solution that don't address the problems of real deployments: coordinating change and responding to it. Updating host files or ssh moduli is about all it seems good for. In reality, files have to be changed at just the right time in a series of coordinated changes with everything else. Chef is what I reach for first, because it solves the problem in the most logical and most complete way. Some people prefer Puppet and that works too. But it's several systems together, like TheForeman (not just Foreman) that make for more powerful infrastructure. There are half a dozen or so other systems in addition to some homegrown (nonportable) ones out there but most others fail to meet requirements Chef and Puppet address (security, auditing, scalability, non/determinism).
Hiera is a like a hierarchal property list that puppet chef etc can use to pull its state. Use zk as a backend to keep multiple instance of hiera in sync to scale for large shops. The neat thing is, like OpenLDAP, Hiera has multiple backends to store your single source of truth. It's like if you wanted to run a Heroku, the environment variables for apps would go in there. Zookeeper is solid but hard to use directly. A more concrete example is a MySQL cluster: you'd want to know what are the hostnames and what is the write master. Sure you could write something that talks to zookeeper but it wouldn't be very general purpose. It's for the use case of scaling beyond what Redis can handle and you do want to get tied to any particular technology other than Hiera. If anyone has funding lying around and wants to take infrastructure smarter and farther, talk to R.I. He knows what the opportunities are and where it needs to go.
http://projects.puppetlabs.com/projects/hiera
PS: I'm glad we're past the days when shops' Cfengine would have a permissions battle with itself (chmod 4750 -> 750 -> 4750 -> ... )
It's bad to promote a solution that don't address the problems of real deployments: coordinating change and responding to it. Updating host files or ssh moduli is about all it seems good for. In reality, files have to be changed at just the right time in a series of coordinated changes with everything else. Chef is what I reach for first, because it solves the problem in the most logical and most complete way. Some people prefer Puppet and that works too. But it's several systems together, like TheForeman (not just Foreman) that make for more powerful infrastructure. There are half a dozen or so other systems in addition to some homegrown (nonportable) ones out there but most others fail to meet requirements Chef and Puppet address (security, auditing, scalability, non/determinism).
Hiera is a like a hierarchal property list that puppet chef etc can use to pull its state. Use zk as a backend to keep multiple instance of hiera in sync to scale for large shops. The neat thing is, like OpenLDAP, Hiera has multiple backends to store your single source of truth. It's like if you wanted to run a Heroku, the environment variables for apps would go in there. Zookeeper is solid but hard to use directly. A more concrete example is a MySQL cluster: you'd want to know what are the hostnames and what is the write master. Sure you could write something that talks to zookeeper but it wouldn't be very general purpose. It's for the use case of scaling beyond what Redis can handle and you do want to get tied to any particular technology other than Hiera. If anyone has funding lying around and wants to take infrastructure smarter and farther, talk to R.I. He knows what the opportunities are and where it needs to go.
http://projects.puppetlabs.com/projects/hiera
PS: I'm glad we're past the days when shops' Cfengine would have a permissions battle with itself (chmod 4750 -> 750 -> 4750 -> ... )
> Hiera is a like a hierarchal property list that puppet chef etc can use to pull its state
Like Etcd. Or LDAP. Or DNS / Hesiod. Or a directory structure on a network file system. Or about 100 other solutions, many of them decades old (see DNS / Hesiod)
So if you have no arguments for why Etcd is a "bad solution", then why do you claim it is one?
From your little description of Hiera, it doesn't sound like it provides anything particularly interesting for the use-cases I want Etcd for. The appeal of Etcd is simplicity, that Hiera is built on Zookeeper and has multiple backends does not make it appealing.
Like Etcd. Or LDAP. Or DNS / Hesiod. Or a directory structure on a network file system. Or about 100 other solutions, many of them decades old (see DNS / Hesiod)
So if you have no arguments for why Etcd is a "bad solution", then why do you claim it is one?
From your little description of Hiera, it doesn't sound like it provides anything particularly interesting for the use-cases I want Etcd for. The appeal of Etcd is simplicity, that Hiera is built on Zookeeper and has multiple backends does not make it appealing.
I can't figure out what etcd is. Any help?
It can mean different things to different people. I will try to explain how I came to understand how etcd could be valuable for me, which may not be indicative of anyone else's use.
We have lots of configuration automation tools: Chef, Puppet, Ansible, SaltStack, etc. These tools make it easier to orchestrate the modification of configuration files on a node. That is well and good.
However, what is my "source of truth" for how things should be configured? Does the fact that node593958.my.co is my primary database server for app foo belong inside Chef, or another tool, or is it more suited for a "directory" of sorts?
No one can answer that but you. But, if you decide that it shouldn't be embedded in your configuration orchestration tool, you may decide to put it in a key value store like etcd. etcd could expose (via REST), something like:
.../apps/foo/primary_db = node593958.my.co
Configuration management could then key off of that source of truth to configure the application appropriately.
etcd exposes an easy to use REST interface to arbitrary storage of hierarchical keys and values.
Nothing is new under the sun. During the first five minutes I was playing around with etcd, I realized I was effectively reinventing LDAP. This is neither wrong nor right, it's simply a different tool for the job. If you like REST and simple interfaces, it's probably a good tool.
We have lots of configuration automation tools: Chef, Puppet, Ansible, SaltStack, etc. These tools make it easier to orchestrate the modification of configuration files on a node. That is well and good.
However, what is my "source of truth" for how things should be configured? Does the fact that node593958.my.co is my primary database server for app foo belong inside Chef, or another tool, or is it more suited for a "directory" of sorts?
No one can answer that but you. But, if you decide that it shouldn't be embedded in your configuration orchestration tool, you may decide to put it in a key value store like etcd. etcd could expose (via REST), something like:
.../apps/foo/primary_db = node593958.my.co
Configuration management could then key off of that source of truth to configure the application appropriately.
etcd exposes an easy to use REST interface to arbitrary storage of hierarchical keys and values.
Nothing is new under the sun. During the first five minutes I was playing around with etcd, I realized I was effectively reinventing LDAP. This is neither wrong nor right, it's simply a different tool for the job. If you like REST and simple interfaces, it's probably a good tool.
Y'know how you can use ENV on Linux to have programs be configured or tell multiple programs what port something is listening on?
Imagine that, but across multiple servers at once. Plus a few other things that some use cases have been built up for.
Imagine that, but across multiple servers at once. Plus a few other things that some use cases have been built up for.
"A highly-available key value store for shared configuration and service discovery. etcd is inspired by zookeeper and doozer..." -- https://github.com/coreos/etcd
It is a distributed configuration management clustering product like zookeeper. It simplifies the creation and sharing of mostly configuration data in clustered or distributed systems.
I don't know what that means or what zookeeper is... Can you explain the problem it solves?
Your web app needs to know what IP address, username and password to connect to the database on.
If it's all one the same box, that's trivial.
But imagine you have several app serves, and several other systems that all needs to talk to the database. Then one day you need to change details of the database server. Now you can either use a tool to update all the configurations (or do it by hand), or you can have all of these apps pull the database config from a distributed directory of some sort.
Etcd is such a directory. Similar to LDAP, but simpler (LDAP requires you to write schemas, and decide which attributes to index on and in set up replication hierarchies). Some people also use DNS for this (you can put arbitrary data in a private DNS server).
If it's all one the same box, that's trivial.
But imagine you have several app serves, and several other systems that all needs to talk to the database. Then one day you need to change details of the database server. Now you can either use a tool to update all the configurations (or do it by hand), or you can have all of these apps pull the database config from a distributed directory of some sort.
Etcd is such a directory. Similar to LDAP, but simpler (LDAP requires you to write schemas, and decide which attributes to index on and in set up replication hierarchies). Some people also use DNS for this (you can put arbitrary data in a private DNS server).
Imagine you're working with a processing system made up of tens or hundreds of computers all (usually) identically configured, networked together and solving small chunks of a large problem. This is software to minimize the effort required to configure all of those individual systems which make up the "compute cluster".
Zookeeper is a very specialized distributed filesystem for saving and delivering important details for very large systems. Getting concurrent distributed systems like this right is the equivalent of summited Mount Everest starting by walking from Paris. It was mostly developed at Yahoo and spun off as an Apache project under Hadoop and commercialized by Cloudera. Netflix uses ZK for example.
so you are basically saying that you don't trust the people who designed etcd.
It's trying to solve with marketing what Chef does.
There is more to configuration management that templating files, but in inspection and having other services arriving at other states.
There is more to configuration management that templating files, but in inspection and having other services arriving at other states.
It is not even trying to solve remotely the same issues as Chef. For starters, it is not going to install or uninstall packages or modify files, or nearly any of the things that a typical Chef recipe might do.
There's no way we'd rely on a public discovery service, for example. If we're going to hardcode configuration information - such as the URL of a discovery service that may or may not be up or reachable, we might as well hardcode the addresses of a few peers.
And running a second etcd cluster to bring up the main one seems pointless. Either it's turtles all the way down, or you then need to hardcode config information for the second cluster, in which case it serves little purpose.
I'd rather have a mechanism where each peer takes a list of possible peers and tries to connect, with a method for deciding when there is quorum to elect an initial leader and start allowing writes (that's easy enough by introducing a config option to decide if a peer is "blessed" to be part of the initial leadership election, and how many blessed peers must be connected to have quorum - just needs to be enough to form a majority of blessed peers to prevent more than one subset from electing a leader before they manage to connect)
Am I missing something?