Docker libswarm(github.com)
github.com
Docker libswarm
https://github.com/docker/libswarm
5 comments
We introduced libswarm at Dockercon today, along with a few other projects. Here is the presentation: http://www.slideshare.net/shykes/docker-the-road-ahead
Will videos of Dockercon talks be posted?
> Organize your application as loosely coupled services from day 1, without over-engineering.
Seems a little counterintuitive
Seems a little counterintuitive
This alludes to the fact that you can do this within the same process, simply by running goroutines and communicating over Go channels. Later, when you actually need to scale, your services already support the interface necessary to be exposed over unix sockets, websocket or spdy/tls: you just need to change the plumbing.
To do this we take advantage of a low-level communication library called libchan: https://github.com/docker/libchan
To do this we take advantage of a low-level communication library called libchan: https://github.com/docker/libchan
What if most of my services are not written in Go?
What if most of my services (postgres, redis, etc.) are not even written by myself?
What if most of my services (postgres, redis, etc.) are not even written by myself?
1: We plan on porting libswarm to every major language.
2: The idea is to use socket passing, so you can use libswarm for service discovery. Place your postgres service behind a libswarm proxy, which implements the "connect" verb: that allows a client to use libswarm for discovery.
2: The idea is to use socket passing, so you can use libswarm for service discovery. Place your postgres service behind a libswarm proxy, which implements the "connect" verb: that allows a client to use libswarm for discovery.
What does libswarm offer over unix sockets?
Is the idea for the project to be a rallying point to normalize protocols for different services that already talk over unix sockets?
Is the idea for the project to be a rallying point to normalize protocols for different services that already talk over unix sockets?
Blog post interview with Mr Hykes on libswarm can be found here http://bit.ly/libswarm
Hi all, a quick warning: this is very rough and early, and not yet ready for production use.
What we're aiming for is Unix-style composition. Because libswarm services communicate over unidirectional message streams (similarly to how goroutines communicate over channels [1]), they can be combined in a way similar to unix pipelines. This makes composition much easier, and is an elegant solution to the problem of hardcoded dependencies in service-oriented architectures (for service A to combine service B a C, it needs to know specifically B and C. Instead what you want the logic of A to be decoupled and re-usable with another component D, without changing it. That's what unix-style composition allows).
TLDR: libswarm allows network services to be composed like unix commands. But it's still very early!
[1] libswarm is built on a small library called libchan: https://github.com/docker/libchan
What we're aiming for is Unix-style composition. Because libswarm services communicate over unidirectional message streams (similarly to how goroutines communicate over channels [1]), they can be combined in a way similar to unix pipelines. This makes composition much easier, and is an elegant solution to the problem of hardcoded dependencies in service-oriented architectures (for service A to combine service B a C, it needs to know specifically B and C. Instead what you want the logic of A to be decoupled and re-usable with another component D, without changing it. That's what unix-style composition allows).
TLDR: libswarm allows network services to be composed like unix commands. But it's still very early!
[1] libswarm is built on a small library called libchan: https://github.com/docker/libchan
How does libswarm compare to etcd/fleet?
The author of Fleet have started contributing a Fleet adapter :)
libswarm exposes a standard interface to systems like Fleet, Mesos, Consul etc. It also has adapters for popular infrastructure providers, and for the Docker API itself. That way you can compose your architecture from standard building blocks, and change the building blocks over time without being locked in.
There was huge demand for this in the Docker system because there are so many orchestration tools being announced and competing for the attention of developers and sysadmins. Just at Dockercon, New Relic announced "Centurion", Spotify announced "Helios", Google announced "Kubernetes". Having a standard interface allows for these tools to continue competing without the risk of fragmentation and lock-in.
Of course the interface has to be thin enough to not get in the way. That's why we're focusing on keeping the codebase small, and the API as simple as possible.
libswarm exposes a standard interface to systems like Fleet, Mesos, Consul etc. It also has adapters for popular infrastructure providers, and for the Docker API itself. That way you can compose your architecture from standard building blocks, and change the building blocks over time without being locked in.
There was huge demand for this in the Docker system because there are so many orchestration tools being announced and competing for the attention of developers and sysadmins. Just at Dockercon, New Relic announced "Centurion", Spotify announced "Helios", Google announced "Kubernetes". Having a standard interface allows for these tools to continue competing without the risk of fragmentation and lock-in.
Of course the interface has to be thin enough to not get in the way. That's why we're focusing on keeping the codebase small, and the API as simple as possible.
fleet and etcd are orthogonal to the problems that libswarm is trying to solve.
libswarm has a future as a nice way of doing composition of networked services that are represented as objects in pipelines. The first demo is a `swarmd` daemon that exposes a docker API endpoint on one end and aggregates between a lot of "real" Docker endpoints.
The big open question is how this sort of composition works when the objects have different interfaces. So, how do we expose a "swarm" API that can be shared between different consistent data stores like etcd, zookeeper or consul. Or how do we encapsulate things like a runtime configured load balancer like nginx, apache2 or vulcand using a unified interface.
My hope for the project is to look at the interface model of Go or the DBus protocol and try to get consensus around the interfaces that make sense and hide complexity.
libswarm has a future as a nice way of doing composition of networked services that are represented as objects in pipelines. The first demo is a `swarmd` daemon that exposes a docker API endpoint on one end and aggregates between a lot of "real" Docker endpoints.
The big open question is how this sort of composition works when the objects have different interfaces. So, how do we expose a "swarm" API that can be shared between different consistent data stores like etcd, zookeeper or consul. Or how do we encapsulate things like a runtime configured load balancer like nginx, apache2 or vulcand using a unified interface.
My hope for the project is to look at the interface model of Go or the DBus protocol and try to get consensus around the interfaces that make sense and hide complexity.
This is a layer "above" etcd/fleet/geard/shipyard/etc.
The idea is to standardize the way that administrators/users/developers "talk" to the various underlying containers. The intent is for it to have plugin support for etcd and fleet and all the other tools in that layer.
The idea is to standardize the way that administrators/users/developers "talk" to the various underlying containers. The intent is for it to have plugin support for etcd and fleet and all the other tools in that layer.