The Curious Case of Linux Containers(medium.com)
medium.com
The Curious Case of Linux Containers
https://medium.com/@sumbry/the-curious-case-of-linux-containers-328e2adc12a2#.ehqvuvenq
26 comments
Someone once said that the public cloud was never about virtualization but rather automation. I would contend that the container revolution we're going through now isn't really about containers but ultimately distributed system platforms. And if it changes the way many have traditionally approached problems great!
My main point is that we're really just at the beginning of all of this. I'm not picking on Docker (I'm actually embracing containers) but rather pointing out that many of these next gen solutions are only tackling the basic use cases right now. While I'm excited there is so much more work left to be done. Implementing many of these solutions still requires a bunch of engineering effort and I'd like to see more turnkey solutions. The developer end is definitely getting easier and more productive but the operational end is getting more complex and still not solving some use cases everyone has. How many different ways can different companies implement HA MySQL for example?!
There are tons of other platforms out there that have recognized and solved many of this class of problems for years but the Cloud and microservices are actually starting to make this worse as adoption skyrockets. Platforms are not really a new thing, we went through this with JVM Platforms a ~decade ago, Heroku style PaaS ~five years ago, and now containers + cloud today.
I guess this is what progress looks like :)
My main point is that we're really just at the beginning of all of this. I'm not picking on Docker (I'm actually embracing containers) but rather pointing out that many of these next gen solutions are only tackling the basic use cases right now. While I'm excited there is so much more work left to be done. Implementing many of these solutions still requires a bunch of engineering effort and I'd like to see more turnkey solutions. The developer end is definitely getting easier and more productive but the operational end is getting more complex and still not solving some use cases everyone has. How many different ways can different companies implement HA MySQL for example?!
There are tons of other platforms out there that have recognized and solved many of this class of problems for years but the Cloud and microservices are actually starting to make this worse as adoption skyrockets. Platforms are not really a new thing, we went through this with JVM Platforms a ~decade ago, Heroku style PaaS ~five years ago, and now containers + cloud today.
I guess this is what progress looks like :)
Ha you weren't kidding about the number of HA MySQL drop-in engine replacements out there, damn. At least most people follow the ANSI-SQL standard, which is something.
I'm not being snarky, but what are the use-cases that everyone have? I'd argue authentication and authorization would be one of the few things, and it was already solved with JSR 196 like 15 years ago. It worked, had a well defined spec, and most importantly it had interop with PAM on UNIX, Kerberos, LDAP, raw db's, mixed mode, you name it. Everyone complained it was too enterprisey and no one used it, so they re-invented the wheel in the early 00s with Rails and Devise and Authlogic and a billion other non-standards. Transactions? Persistence? Java's spec took care of that too, and fairly rigorously. So I'm with 100% with you on this being a solved problem.
We're making gradual progress (i.e., the option to develop in a language where the correctness of our code can be formally verified thanks to more readily available, mathematically sound type-systems) but like any society there are trends, and where there are trends you'll have the recurrence of many old things (Interpol ripping off Joy Division) and the invention of a few new things (where are often the composition of two older ideas, or the implementation of a new idea which wasn't computationally feasible previously but now is, or a concept from another industry like signal analysis or three-phase road traffic theory applied to our code-monkey'ing domain).
RE: Overall progress - Microsoft is doing some fantastic things in Powershell, effectively taking concepts like package management, man pages, and the shell, extracting the best elements from each of those, and implementing them in a consistent manner. No more choosing between systemd or init.d or other holy wars. If you want to do it differently, you effectively have a standardized interface to write your new implementation against within most of the platform. Don't like ASP.NET's templating system? No problem, it's all open source, and you can swap your own in, but it's all modular so nothing will break, and your co-workers can continue working in the traditional Razor templating.
I'm not being snarky, but what are the use-cases that everyone have? I'd argue authentication and authorization would be one of the few things, and it was already solved with JSR 196 like 15 years ago. It worked, had a well defined spec, and most importantly it had interop with PAM on UNIX, Kerberos, LDAP, raw db's, mixed mode, you name it. Everyone complained it was too enterprisey and no one used it, so they re-invented the wheel in the early 00s with Rails and Devise and Authlogic and a billion other non-standards. Transactions? Persistence? Java's spec took care of that too, and fairly rigorously. So I'm with 100% with you on this being a solved problem.
We're making gradual progress (i.e., the option to develop in a language where the correctness of our code can be formally verified thanks to more readily available, mathematically sound type-systems) but like any society there are trends, and where there are trends you'll have the recurrence of many old things (Interpol ripping off Joy Division) and the invention of a few new things (where are often the composition of two older ideas, or the implementation of a new idea which wasn't computationally feasible previously but now is, or a concept from another industry like signal analysis or three-phase road traffic theory applied to our code-monkey'ing domain).
RE: Overall progress - Microsoft is doing some fantastic things in Powershell, effectively taking concepts like package management, man pages, and the shell, extracting the best elements from each of those, and implementing them in a consistent manner. No more choosing between systemd or init.d or other holy wars. If you want to do it differently, you effectively have a standardized interface to write your new implementation against within most of the platform. Don't like ASP.NET's templating system? No problem, it's all open source, and you can swap your own in, but it's all modular so nothing will break, and your co-workers can continue working in the traditional Razor templating.
I'm quite new to containers, but it seems to me that Docker revolutionizes sysadmin the same way npm revolutionizes Javascript. It's not that it magically makes your problems any easier to solve, it's that it lets you declare your solution in a formal way without unstated dependencies, and then distribute your solution in a way that lets others declare a dependency on it.
(More broadly I feel like this same thing has been happening in various parts of the IT landscape over the past few years, to the extent that their ought to be a name for it, if there isn't already?)
(More broadly I feel like this same thing has been happening in various parts of the IT landscape over the past few years, to the extent that their ought to be a name for it, if there isn't already?)
What I find curious about all the container discussions and narrative is the strange lack of context. Sure discuss Docker but also discuss namespaces, cgroups, overlayfs, aufs, and all the other critical enabling technologies where a lot of major problems with containers exist and will be solved. For instance user namespaces, cgroups are not namespace aware, how to integrate overlayfs or aufs so they can be mounted by user namespaces seamlessly.
Surely these projects and developers need support and focus. Or else it become mere marketing for companies that have the funds or ability to market themselves. Do we just talk about libvirt without context or understanding of kvm and xen, how would that be useful or meaningful?
An ‘immutable container’ is nothing but launching a copy of a container enabled by overlay file systems like aufs or overlayfs, a ‘stateless’ container is a bind mount to the host. Using worlds like stateless, immutable or idempotent just obscures simple underlying technologies and prevents wider understanding of core Linux technologies that need to be highlighted and supported. How is this a sustainable development model?
Docker chooses to run containers without an init. The big problem here is most if not all apps you want to run in a container are not designed to work in an init less environment and require daemons, services, logging, cron and when run beyond a single host, ssh and agents. This adds a boatload of additional complexity for users before you can even deploy your apps, and a lot of effort is expended is just managing the basic process of running apps and managing their state in this context.
Contrast that with LXC containers which have a normal init and can manage multiple processes enabling for instance your VM workloads to move seamlessly to containers without any extra engineering. Any orchestration, networking, distributed storage you already use will work obviating the need for reinventing. That’s a huge win and a huge use case that makes deployment simple and slices all the complexity, but if you listen to the current container narrative and the folks pushing a monoculture and container standards it would appear there are no alternatives and running init less containers is the only ‘proper’ way to use containers, never mind the additional complexity may only make sense for specific use cases.
Surely these projects and developers need support and focus. Or else it become mere marketing for companies that have the funds or ability to market themselves. Do we just talk about libvirt without context or understanding of kvm and xen, how would that be useful or meaningful?
An ‘immutable container’ is nothing but launching a copy of a container enabled by overlay file systems like aufs or overlayfs, a ‘stateless’ container is a bind mount to the host. Using worlds like stateless, immutable or idempotent just obscures simple underlying technologies and prevents wider understanding of core Linux technologies that need to be highlighted and supported. How is this a sustainable development model?
Docker chooses to run containers without an init. The big problem here is most if not all apps you want to run in a container are not designed to work in an init less environment and require daemons, services, logging, cron and when run beyond a single host, ssh and agents. This adds a boatload of additional complexity for users before you can even deploy your apps, and a lot of effort is expended is just managing the basic process of running apps and managing their state in this context.
Contrast that with LXC containers which have a normal init and can manage multiple processes enabling for instance your VM workloads to move seamlessly to containers without any extra engineering. Any orchestration, networking, distributed storage you already use will work obviating the need for reinventing. That’s a huge win and a huge use case that makes deployment simple and slices all the complexity, but if you listen to the current container narrative and the folks pushing a monoculture and container standards it would appear there are no alternatives and running init less containers is the only ‘proper’ way to use containers, never mind the additional complexity may only make sense for specific use cases.
Docker chooses to run containers without an init.
I don't think Docker chooses one way or the other, and people do run Docker with an init: http://phusion.github.io/baseimage-docker/
I don't think Docker chooses one way or the other, and people do run Docker with an init: http://phusion.github.io/baseimage-docker/
Correct, Docker doesn't care what you run inside the container. You provide a command to run, and it runs it. That command may be your application server or a traditional init process which in turn will fork multiple children.
Docker does make it easier to follow an "application container" pattern, and that pattern avoids (with good reason) booting an entire traditional init system inside the container. But following that pattern is not mandatory. Not forcing too many patterns upon users all at once was part of the original Docker philosophy. Unfortunately that aspect was drowned in the cacophony as a few loud and passionate people interpreted Docker through the lens of their own favorite patterns.
In retrospect I wish we had been more assertive in reminding everyone to respect the philosophy behind Docker: that you can share tools with people without forcing everyone to use them in the same way as you.
Docker does make it easier to follow an "application container" pattern, and that pattern avoids (with good reason) booting an entire traditional init system inside the container. But following that pattern is not mandatory. Not forcing too many patterns upon users all at once was part of the original Docker philosophy. Unfortunately that aspect was drowned in the cacophony as a few loud and passionate people interpreted Docker through the lens of their own favorite patterns.
In retrospect I wish we had been more assertive in reminding everyone to respect the philosophy behind Docker: that you can share tools with people without forcing everyone to use them in the same way as you.
Prominent Docker, Inc., employees have spent years arguing against it, and they don't prioritize bugs around that use case. Which matters.
My favourite is Alpine Linux with s6-overlay: https://github.com/just-containers/s6-overlay
Overlayfs, aufs, etc. are really irrelevant to containers. They are used in Docker because it's built around opaque binary disk images that do not compose, which I believe is a big flaw, but it doesn't have to be this way. For example, GNU Guix has a container implementation that does not use disk images at all but still achieves system-wide deduplication of software across all containers running on the host via simple bind mounts.
LXC containers which have a normal init
Incorrect. You can do custom application (eg. single-process) or init-based containers with LXC.
Incorrect. You can do custom application (eg. single-process) or init-based containers with LXC.
I think in discussions its useful to deal with defaults because there are always workarounds. For Docker the default is to run init less containers so you do not get a normal OS environment. For LXC the default is run init in the container so you get a container with a normal OS environment out of the box, more like a VM.
lxc-start by default will start the container's init and any apps installed with services will start with container start. lxc-start has no options or documentation I have seen that enables users to launch as a single process without starting the container init so it will be useful to share how that works.
With Docker you launch the app in the container directly from the host and if the app is a daemon for instance Nginx, you need to disable daemon mode or it won't work as there is no init process to manage background processes. So with Nginx you would need to disable daemon mode in nginx.conf or start it with 'nginx -g "Daemon off"' in Docker. In contrast with LXC you install Nginx and it will work as it would on bare metal or VMs. There is no need to think about how the app manages its processes
To run multiple processes or daemons in Docker you need to use a shell script, or use a third party process manager like supervisord for instance. The entire Docker ecosystem, and tooling is built and defined around single process containers.
A ton of problems around deployments especially for multi process apps, daemons, databases or any apps that require cron, logging, agents etc emanate here. This becomes a process in itself to know how the apps operates and then to configure it accordingly. Your deployments also become docker specific, for instance Nginx with a Wordpress stack deployed on bare metal or VM can move seamlessly to an LXC container and vice versa because they all run normal OS environments.
This can't happen with Docker and you need to re-engineer deployments. Why take on this extra load, and create incompatibility from a non standard OS environment, or worse give up the init that comes for free for for a third party process manager to do the exact same thing, only with more effort and cognitive load? This doesn't make much sense unless there is an extremely clear upside and use case.
lxc-start by default will start the container's init and any apps installed with services will start with container start. lxc-start has no options or documentation I have seen that enables users to launch as a single process without starting the container init so it will be useful to share how that works.
With Docker you launch the app in the container directly from the host and if the app is a daemon for instance Nginx, you need to disable daemon mode or it won't work as there is no init process to manage background processes. So with Nginx you would need to disable daemon mode in nginx.conf or start it with 'nginx -g "Daemon off"' in Docker. In contrast with LXC you install Nginx and it will work as it would on bare metal or VMs. There is no need to think about how the app manages its processes
To run multiple processes or daemons in Docker you need to use a shell script, or use a third party process manager like supervisord for instance. The entire Docker ecosystem, and tooling is built and defined around single process containers.
A ton of problems around deployments especially for multi process apps, daemons, databases or any apps that require cron, logging, agents etc emanate here. This becomes a process in itself to know how the apps operates and then to configure it accordingly. Your deployments also become docker specific, for instance Nginx with a Wordpress stack deployed on bare metal or VM can move seamlessly to an LXC container and vice versa because they all run normal OS environments.
This can't happen with Docker and you need to re-engineer deployments. Why take on this extra load, and create incompatibility from a non standard OS environment, or worse give up the init that comes for free for for a third party process manager to do the exact same thing, only with more effort and cognitive load? This doesn't make much sense unless there is an extremely clear upside and use case.
I agree, docker is half-baked.
> Docker chooses to run containers without an init. The big problem here is most if not all apps you want to run in a container are not designed to work in an init less environment and require daemons, services, logging, cron and when run beyond a single host, ssh and agents.
Do this not confuse init with a process supervisor?
Do this not confuse init with a process supervisor?
A process supervisor (foreman, monit, etc. ) is just one of the things an init system starts. The set of items you want to ensure are always running isn't usually the same as the set of items you want to start with the system.
Many of the author's desires are handled in SmartOS (https://www.joyent.com/blog/triton-docker-and-the-best-of-al...)
Totally agreed. I don't like Docker but I do like both SmartOS and FreeBSD jails. If you're even somewhat interested in containers you need to take a look at SmartOS (I liked http://timboudreau.com/blog/smartos/read as an approachable "getting started") and FreeBSD jails (https://www.freebsd.org/doc/handbook/jails.html). For Jails I like ezjail and iocage as good wrappers so you may want to start there. You can't have a coherent opinion on the technology if you haven't seen how other people are doing it.
And OpenShift: https://enterprise.openshift.com/
I'll pile on!
Cloud Foundry is another opensource PaaS: https://www.cloudfoundry.org/learn/features/
(Disclaimer: I work for Pivotal, which donates the majority of engineering effort to Cloud Foundry)
Cloud Foundry is another opensource PaaS: https://www.cloudfoundry.org/learn/features/
(Disclaimer: I work for Pivotal, which donates the majority of engineering effort to Cloud Foundry)
I don't like containers for the purpose of deployment. All they do is hide complexity from DevOps people - But the complexity is still very much there. Hiding complexity from DevOps people is useful in a PaaS context, but it is an anti-pattern when you consider pretty much every other use case.
Docker encourages developers to keep adding more and more different technologies to a system. If you consider most popular applications today, they are usually made up of hundreds of different technologies - Each of which requires their own bootstrap/setup logic.
Maybe if each micro-service was made up of fewer different technologies, deployment wouldn't be such a headache and you wouldn't need Docker to begin with.
Docker encourages developers to keep adding more and more different technologies to a system. If you consider most popular applications today, they are usually made up of hundreds of different technologies - Each of which requires their own bootstrap/setup logic.
Maybe if each micro-service was made up of fewer different technologies, deployment wouldn't be such a headache and you wouldn't need Docker to begin with.
Author's terminology seems muddled. The first bullet is correct, but only on a technicality, and it fails to point out that strictly there is no "Linux container," only an emergent and weakly cohesive combination of features that as an artifact create a so-called Linux container (as opposed to being a well-defined atomic unit/OS resource). The third bullet equivocates archive formats with application containers.
The author's 'platform' concerns are in fact infrastructure automation concerns generic to any service-oriented architecture and not specific to containers.
My take as an early LXC adopter (way pre docker) and from-concept builder of two production clusters using custom automation:
1. Service state. The author essentially requests master-slave promotion. Corosync/pacemaker is an excellent place to look for well tested solutions here. The normal approach used there is superior to the author's, ie. a floating IP used by all clients is switched to an already live replacement master, which is first jabbed in to action, and which shares the same backing storage as the nodes which has failed. (A great solution for shared backing storage on standard hardware is DRBD)
2. MySQL update. Change management with SOA has to be far more rigid due to complex interdependencies. Typically you version your entire service, exhaustively test it against peer services, and only then deploy the updated service. This implies a workflow process more formalized than the series of manual operations the author hand-waves about. That said, in a typical database scenario it is often possible to upgrade across minor versions simply by updating the binary, since it will read the old on-disk database store fine.
3. Heartbleed. With an appropriately automated approach to service builds, this should be trivial. Mask the affected packages or versions, rebuild all services, test and deploy. This really goes back again to overall workflow formalization and automation. (Solid infrastructure would have cryptographic signoff on stuff before it runs, eg. production demands a signature from test infrastructure that all tests have passed before executing a new service)
4. Service dependencies. This is far more complex than people assume. My advice is to use corosync/pacemaker, a well tested approach. (There are others)
5. Hadoop. Same as any other service.
There are many other problems with SOA-type infrastructure automation; some ideas I wrote up ages ago can be seen at http://stani.sh/walter/pfcts/ as well as the sketch/documentation of the solution I built.
My take as an early LXC adopter (way pre docker) and from-concept builder of two production clusters using custom automation:
1. Service state. The author essentially requests master-slave promotion. Corosync/pacemaker is an excellent place to look for well tested solutions here. The normal approach used there is superior to the author's, ie. a floating IP used by all clients is switched to an already live replacement master, which is first jabbed in to action, and which shares the same backing storage as the nodes which has failed. (A great solution for shared backing storage on standard hardware is DRBD)
2. MySQL update. Change management with SOA has to be far more rigid due to complex interdependencies. Typically you version your entire service, exhaustively test it against peer services, and only then deploy the updated service. This implies a workflow process more formalized than the series of manual operations the author hand-waves about. That said, in a typical database scenario it is often possible to upgrade across minor versions simply by updating the binary, since it will read the old on-disk database store fine.
3. Heartbleed. With an appropriately automated approach to service builds, this should be trivial. Mask the affected packages or versions, rebuild all services, test and deploy. This really goes back again to overall workflow formalization and automation. (Solid infrastructure would have cryptographic signoff on stuff before it runs, eg. production demands a signature from test infrastructure that all tests have passed before executing a new service)
4. Service dependencies. This is far more complex than people assume. My advice is to use corosync/pacemaker, a well tested approach. (There are others)
5. Hadoop. Same as any other service.
There are many other problems with SOA-type infrastructure automation; some ideas I wrote up ages ago can be seen at http://stani.sh/walter/pfcts/ as well as the sketch/documentation of the solution I built.
> Did you know that even with all the advances with technology the amount of time that a housewife spends maintaining a household today is the exact same as it was in the 1950s!
Unrelated to the core of the article, I know. But it seemed odd to me. Is this true? Or was it a metaphor?
EDIT: did a bit of googling and found a couple[0][1] of not-extremely-academic references that claim that it's less than half... but couldn't find anything more authoritative.
[0] http://www.telegraph.co.uk/women/womens-life/9721147/Women-s...
[1] https://www.anglianhome.co.uk/50years
Unrelated to the core of the article, I know. But it seemed odd to me. Is this true? Or was it a metaphor?
EDIT: did a bit of googling and found a couple[0][1] of not-extremely-academic references that claim that it's less than half... but couldn't find anything more authoritative.
[0] http://www.telegraph.co.uk/women/womens-life/9721147/Women-s...
[1] https://www.anglianhome.co.uk/50years
Are cgroups meant to be secure against malicious code trying to get out and install rootkits on the underlying system?
AFAIK, hardware-level virtualization is. If code running under Xen can get out, that's considered a bug in Xen and it gets fixed, right?
OTOH, chroot is not considered to be secure. It isn't designed to be secure. It is not a security tool. Code in a chroot jail is expected to be able to leave without subverting the security mechanisms which, by and large, don't exist in the first place.
So, are cgroups Xen or chroot?
AFAIK, hardware-level virtualization is. If code running under Xen can get out, that's considered a bug in Xen and it gets fixed, right?
OTOH, chroot is not considered to be secure. It isn't designed to be secure. It is not a security tool. Code in a chroot jail is expected to be able to leave without subverting the security mechanisms which, by and large, don't exist in the first place.
So, are cgroups Xen or chroot?
Containers (which are more than just cgroups) are supposed to be secure.
In practice, the attack surface is much bigger for containers than VMs, so there are far more "break out of container" vulnerabilities out there than "break out of VM" vulnerabilities.
Most containers are run in a VM, usually one being run for you by Amazon or Google.
In practice, the attack surface is much bigger for containers than VMs, so there are far more "break out of container" vulnerabilities out there than "break out of VM" vulnerabilities.
Most containers are run in a VM, usually one being run for you by Amazon or Google.
"Also be warned — I have no idea what I’m talking about. Take everything in this article with a grain of salt." - pretty much. I'm not sure whether the author thinks "platforms" even are. And the complaints that "linux cgroups aren't the entire platform so why call it containers" seem pedantic.
I don't see why "handling MySQL failover", "resource management", "upgrading OpenSSL" should all be handled by the same piece of software in any case.
I don't see why "handling MySQL failover", "resource management", "upgrading OpenSSL" should all be handled by the same piece of software in any case.
It's easy to point at Docker and say it's a trivial wrapper around some technologies that have been around for years and call everyone crazy for buying into the 'Docker is going to change the (sysadmin) world' hype. Of course in essence it's pretty trivial, many things are.
Docker, and Docker-like systems really are changing the sysadmin world. Easily formalised software environments is what this hype is about, and the fact that the Docker community isn't really looking at the stateful service problem right, doesn't mean they're not enabled by Docker either.
To me, and to many 'devops' (i.e. developers rolling into sysadmin roles) Docker was eye-opening. We reduced our chef scripts by 80%, and thus our headaches by 95%. It's saving us months of ops work. If you were a seasoned sysadmin and rolling your eyes at the fact that we were rolling out application level services in the same chef codebase we were setting up our system environments with, then yeah you already were better than us. But as far as I could tell the way we used to do it really was the status quo, at least in the startup/small it business world.