Docker joins the Linux Foundation(blog.docker.io)
blog.docker.io
Docker joins the Linux Foundation
http://blog.docker.io/2013/06/dotcloud-and-docker-join-the-linux-foundation/
4 comments
Here is a 2006 article about it - there is no doubt more stuff on LWN if you dig around https://lwn.net/Articles/179361/
The gist is that there were a lot of out of tree solutions (openvz is still used eg by Travis CI) but they had conflicting requirements and a lot of the suggestions were just "we need BSD jails" which is a very monolithic single purpose solution, like having a web server in the kernel (yes there was one) rather than having the kernel APIs in place so that was not needed.
What we have ended up with (mainly cgroups and namespacing basically) is much more flexible in that each piece is for a separate thing, eg you can just namespace networks without full isolation (which is really useful for eg testing network code without messing with the real config).
It did take a very very long time though.
The gist is that there were a lot of out of tree solutions (openvz is still used eg by Travis CI) but they had conflicting requirements and a lot of the suggestions were just "we need BSD jails" which is a very monolithic single purpose solution, like having a web server in the kernel (yes there was one) rather than having the kernel APIs in place so that was not needed.
What we have ended up with (mainly cgroups and namespacing basically) is much more flexible in that each piece is for a separate thing, eg you can just namespace networks without full isolation (which is really useful for eg testing network code without messing with the real config).
It did take a very very long time though.
I've been running OpenVZ for years - It works very well.
LXC is built into the kernel directly, rather than a patch, which is very nice.
So far as I can see, Docker is just some nice wrapper scripts around LXC, nothing particularly revolutionary in and of itself.
LXC is built into the kernel directly, rather than a patch, which is very nice.
So far as I can see, Docker is just some nice wrapper scripts around LXC, nothing particularly revolutionary in and of itself.
Correct, docker is not an implementation of linux containers. The heavy lifting is done by lxc (and alternatives like openvz).
What docker brings is a way for backend developers to package and distribute their app into portable, sandboxed containers (think "iphone apps for the server").
This is only possible now that linux has decent container support, because it means the vast majority of servers out there can potentially run a docker container unmodified.
What docker brings is a way for backend developers to package and distribute their app into portable, sandboxed containers (think "iphone apps for the server").
This is only possible now that linux has decent container support, because it means the vast majority of servers out there can potentially run a docker container unmodified.
Linux has had containers for ages. Docker has simply made them easier to use as a service.
You should be asking why Solaris/BSD never had something as useful as docker despite having zones/jails. And linux supports overlay filesystems (to clone containers cheaply) without the need for ZFS.
You should be asking why Solaris/BSD never had something as useful as docker despite having zones/jails. And linux supports overlay filesystems (to clone containers cheaply) without the need for ZFS.
Here is my conjecture from a few days ago: https://news.ycombinator.com/item?id=5901649
In summary, I think it's because containers, although open source, are a relatively commercial technology. Interested in any comments from people were there in the 90's when lots of hosting providers started using Linux.
In summary, I think it's because containers, although open source, are a relatively commercial technology. Interested in any comments from people were there in the 90's when lots of hosting providers started using Linux.
raises a hand
In the early 2000s, I was in the hosting business, and we started using UML (user-mode-linux), then vserver and openvz. When Xen came around, we switched to that, because it made more sense for us (and for our customers): more control, more "this is your machine" feeling.
"Yes, but containers are better because they are lightweight!" Of course, but this idea was pointless back then, because people who wanted really lightweight things went to shared hosting (and for massive PHP hosting, containers can't beat a single Apache instance serving tens of thousands of vhosts; at least not yet).
Maybe (maybe!) things changed with the rise of stacks like Python/Ruby/Node.js/etc for which there are no cheap shared hosting solutions like there is for PHP. Then containers are obviously cheaper than running your own VPS, and easier to setup (and isolate) than shared hosting on a single box.
In the early 2000s, I was in the hosting business, and we started using UML (user-mode-linux), then vserver and openvz. When Xen came around, we switched to that, because it made more sense for us (and for our customers): more control, more "this is your machine" feeling.
"Yes, but containers are better because they are lightweight!" Of course, but this idea was pointless back then, because people who wanted really lightweight things went to shared hosting (and for massive PHP hosting, containers can't beat a single Apache instance serving tens of thousands of vhosts; at least not yet).
Maybe (maybe!) things changed with the rise of stacks like Python/Ruby/Node.js/etc for which there are no cheap shared hosting solutions like there is for PHP. Then containers are obviously cheaper than running your own VPS, and easier to setup (and isolate) than shared hosting on a single box.
There is OpenVZ, LXC and Linux-VServer which are the same as FreeBSD jails but for the Linux kernel.
There was also another predcessor, UML(User Mode Linux). This was widely available and offered such as by Linode before they moved to Xen.
UML is a little different, using a Linux kernel which relies on the system calls of a parent kernel. The other technologies you mentioned, and LXC which docker is built on, have a single unified kernel which interprets paths and other namespaces in system calls from different process trees differently.
The facilities in the kernel for doing this have gotten better progressively over the years. Support for separate namespaces, and adding more and more of those types of namespaces have only been added in the last few years.
Just a guess:
Vendors certainly seemed more focussed on getting people to use complicated virtualization schemes with proprietary hypervisors. I wonder if it was more a matter of man-power at that point.
In comparison, Sun was already selling systems so the value of selling virtualization separately so zones may have ended up being an elegant integrated solution. I'm not sure of the FreeBSD jail history but there may be a similar story to tell if you look at the source of the contributions.
Vendors certainly seemed more focussed on getting people to use complicated virtualization schemes with proprietary hypervisors. I wonder if it was more a matter of man-power at that point.
In comparison, Sun was already selling systems so the value of selling virtualization separately so zones may have ended up being an elegant integrated solution. I'm not sure of the FreeBSD jail history but there may be a similar story to tell if you look at the source of the contributions.
EDIT: I had previously asked what exactly Docker is.
https://www.youtube.com/watch?feature=player_embedded&v=wW9C... gives a super quick overview. Very interesting technology.
https://www.youtube.com/watch?feature=player_embedded&v=wW9C... gives a super quick overview. Very interesting technology.
Yes, that's exactly the goal of docker: "build once, actually run anywhere".
Docker is a portable container engine. It gives you, the developer, a way to package your app and all its dependencies into a portable container which can be deployed on any modern Linux server: local VM, bare metal server, EC2 instance, doesn't matter. Containers are completely sandboxed and do not interfere with each other (think "iPhone apps for the server").
The specificities of each machine (location, distro, storage, networking) are abstracted away so you can focus on what your application requires (a known filesystem state, tcp ports to receive connections...) without having to worry about how those requirements will be met on each individual machine. This abstraction is based on common unix concepts - files, processes, tcp connections - and is therefore not language-specific, so it doesn't matter what programming language your application is written in.
To do this, Docker relies on a recent feature of the linux kernel called lxc. That's the part which allows sandboxing of applications from one another. The concept itself is not new - what is new is that it is now part of the mainstream kernel, which means almost every server in the world will soon be capable of running docker containers - hence the exciting possibility of "run anywhere".
Docker is a portable container engine. It gives you, the developer, a way to package your app and all its dependencies into a portable container which can be deployed on any modern Linux server: local VM, bare metal server, EC2 instance, doesn't matter. Containers are completely sandboxed and do not interfere with each other (think "iPhone apps for the server").
The specificities of each machine (location, distro, storage, networking) are abstracted away so you can focus on what your application requires (a known filesystem state, tcp ports to receive connections...) without having to worry about how those requirements will be met on each individual machine. This abstraction is based on common unix concepts - files, processes, tcp connections - and is therefore not language-specific, so it doesn't matter what programming language your application is written in.
To do this, Docker relies on a recent feature of the linux kernel called lxc. That's the part which allows sandboxing of applications from one another. The concept itself is not new - what is new is that it is now part of the mainstream kernel, which means almost every server in the world will soon be capable of running docker containers - hence the exciting possibility of "run anywhere".
> Yes, that's exactly the goal of docker: "build once, actually run anywhere".
Except doesn't it only run on Ubuntu? I looked at it to see if it might be useful for my servers, which are running Debian Wheezy, but the "Other Linux" instructions on the Docker site amount to "install VirtualBox and run Ubuntu" — which, yeah, anything will run anywhere if you just run it in a VM, but that was already true.
Except doesn't it only run on Ubuntu? I looked at it to see if it might be useful for my servers, which are running Debian Wheezy, but the "Other Linux" instructions on the Docker site amount to "install VirtualBox and run Ubuntu" — which, yeah, anything will run anywhere if you just run it in a VM, but that was already true.
That's just our docs being unclear :) You can run docker on debian wheezy, no problem.
Docker is currently easier to run on certain distros than others. But it's not because it's distro-specific. It's because it has kernel requirements which are more easily met on certain distros than others. Those requirements are described at http://docs.docker.io/en/latest/installation/kernel/ (short version: aufs + lxc + (<=2.6 OR >=3.8))
Docker is currently easier to run on certain distros than others. But it's not because it's distro-specific. It's because it has kernel requirements which are more easily met on certain distros than others. Those requirements are described at http://docs.docker.io/en/latest/installation/kernel/ (short version: aufs + lxc + (<=2.6 OR >=3.8))
Clean up the docs please.
Even just a list of software and kernel deps, then instructions on how to compile the rest of the software by hand would suffice.
For now I've had to muck around reading the Ubuntu packaging files to get a proper idea what/how you install this thing. Which isn't cool. :-(
Even just a list of software and kernel deps, then instructions on how to compile the rest of the software by hand would suffice.
For now I've had to muck around reading the Ubuntu packaging files to get a proper idea what/how you install this thing. Which isn't cool. :-(
The main blocker is AUFS, but Docker has been shown to run fine with overlayfs and BTRFS; and the latter is available on virtually all recent Linux systems (it's even usable on RHEL kernels, which aren't really known for being bleeding-edge).
The hardest part is to provide interchangeable, pluggable storage systems.
The hardest part is to provide interchangeable, pluggable storage systems.
Basically. One thing to consider is the fact that decoupling applications and services they interact with is generally considered a good practice. What services does a web application need? Generally speaking, a web process to handle requests, a database, and maybe some in-memory storage such as memcache. What docker can do is script the creation of these services as containers that can be deployed "anywhere." So say you have your web app now, and your 3 services are on a single server and the load is getting heavy because of database queries. Docker makes it extremely easy to reliably move the services between servers to handle the load better.
Albeit this is a very simplified vision, I don't think the complete use cases for docker have been hashed out quite yet. tl;dr dev ops is about to get magnitudes easier.
Albeit this is a very simplified vision, I don't think the complete use cases for docker have been hashed out quite yet. tl;dr dev ops is about to get magnitudes easier.
I'm glad that dotCloud is contributing money to the Linux Foundation. I'm curious how involved they are with linux kernel and plumbing development. It seems like the changes coming to cgroups, for example, could be a big deal for Docker.
https://lwn.net/Articles/555920/
https://lwn.net/Articles/555920/
sweet! does this mean that AUFS will be included in tree?
That would be fantastic. Unfortunately I don't have the answer right now.
Have you considered using OverlayFS in a future version of Docker, instead of AUFS? It comes bundled with Ubuntu 12.04 and higher, and my understanding is that it could possibly get merged w/ kernel 3.10
Overlayfs has a static limit of 1 ro layer, which makes it useless out of the box for docker. It's also less proven in production than aufs. Aufs works great and has no decent alternative.
Merging aufs into mainline feels like the logical thing to do, but clearly someone in the chain disagrees. It's disappointing but it is what it is.
Btrfs seems like a more likely alternative for docker.
Merging aufs into mainline feels like the logical thing to do, but clearly someone in the chain disagrees. It's disappointing but it is what it is.
Btrfs seems like a more likely alternative for docker.
I mean, congratulation guys, but why the hell didn't this happen earlier? I know I'm not the brightest bulb in the room but this has been a fundamentally dangerous feature to be missing. I don't get how anyone pretends that their Linux boxes are secure without some extremely personal contrived creations.