Why is Docker the new craze in virtualization and cloud computing?(opensource.com)
opensource.com
Why is Docker the new craze in virtualization and cloud computing?
https://opensource.com/business/14/7/why-docker-new-craze-virtualization-and-cloud-computing
14 comments
They keep saying Vagrant and Docker are two different things but isn't that the whole point with Vagrant?
Yes! But using Docker can help several problems:
First time provisioning of a box can take up to 30 minutes even with a non-mature app. Mature apps with many moving parts can easily get near an hour- using Docker, you can bring that down to 5 minutes or less, with the majority of that being downloading the containers.
Neat thing though, is that you can use Docker WITH Vagrant to do this. The setup I'm doing on my side projects, essentially goes like this:
If I'm developing, I use Vagrant to spin up Docker containers(using Docker as the provisioner) for supporting items- databases, memcached, things like elasticsearch or a message queue- then develop from my machine(a mac). If I want to get someone else in on the project, I don't have to care one iota about what they're on, Docker will make it work.
On top of that, there's the capability to have your app itself running in Docker containers for development. This means no having to worry about dependencies, environment variables, etc- and if you keep things up to date, you can even just upload your app to a private Docker repo, and use that to deploy to your server(though in a team this may not be the most realistic option at the moment, I've not given it a shot, though I don't see why it wouldn't work).
For me, Vagrant is the tool I use to tie everything together for development. Docker is what I have it run off of, instead of Chef/Puppet/Salt/Ansible, so that my build times go from 30 minutes to whatever download speed I get.
First time provisioning of a box can take up to 30 minutes even with a non-mature app. Mature apps with many moving parts can easily get near an hour- using Docker, you can bring that down to 5 minutes or less, with the majority of that being downloading the containers.
Neat thing though, is that you can use Docker WITH Vagrant to do this. The setup I'm doing on my side projects, essentially goes like this:
If I'm developing, I use Vagrant to spin up Docker containers(using Docker as the provisioner) for supporting items- databases, memcached, things like elasticsearch or a message queue- then develop from my machine(a mac). If I want to get someone else in on the project, I don't have to care one iota about what they're on, Docker will make it work.
On top of that, there's the capability to have your app itself running in Docker containers for development. This means no having to worry about dependencies, environment variables, etc- and if you keep things up to date, you can even just upload your app to a private Docker repo, and use that to deploy to your server(though in a team this may not be the most realistic option at the moment, I've not given it a shot, though I don't see why it wouldn't work).
For me, Vagrant is the tool I use to tie everything together for development. Docker is what I have it run off of, instead of Chef/Puppet/Salt/Ansible, so that my build times go from 30 minutes to whatever download speed I get.
Your username is the same as my (jindo) dog! So I'll reply to you.
I've been using vagrant to fix exactly the problem of "works on my box" for years. Docker is way too much to explain to my small development team. They struggled to deal with vagrant for some time before figuring it out.
Vagrant is perfect for us right now, and it would take a force of nature for me to even begin to push docker on them.
I've been using vagrant to fix exactly the problem of "works on my box" for years. Docker is way too much to explain to my small development team. They struggled to deal with vagrant for some time before figuring it out.
Vagrant is perfect for us right now, and it would take a force of nature for me to even begin to push docker on them.
I believe there are plans for offering a Vagrant api to Docker and vice-versa.
In many cases, Docker can be treated like a more lightweight Vagrant. One important benefit I get is more efficient use of machine resources. Running 5-10 fully virtualized Vagrant instances on my laptop hurts. Running the same number of Docker containers is trivial.
The ability to version your images and easily build on them is nice too.
The ability to version your images and easily build on them is nice too.
Run ALL the distributions and versions in prod? :)
I'd rather have the opportunity to re-test and re-deploy one component at a time than be forced to do "big bang upgrades".
In the long run I hope we start seeing distros or tools targeted specifically at building minimal Docker images. E.g. if I'm going to deploy a Ruby based web app in Docker, the stuff I need present in the container is _very_ limited.
In the long run I hope we start seeing distros or tools targeted specifically at building minimal Docker images. E.g. if I'm going to deploy a Ruby based web app in Docker, the stuff I need present in the container is _very_ limited.
No, if your environment is standardized and stripped down you can run something that is very close to prod in your dev environment.
Note docker isn't the only answer for this, for instance, you can use ordinary VMs with vagrant, but docker does it at ludicrous speed.
Note docker isn't the only answer for this, for instance, you can use ordinary VMs with vagrant, but docker does it at ludicrous speed.
So, basically to achieve this you need to run production env using docker too? So that docket could match it.
Also, production data has to be copied over separately?
Are these correct assumptions?
PS: I'm still trying to understand how docket/containers may fit in current process.
Also, production data has to be copied over separately?
Are these correct assumptions?
PS: I'm still trying to understand how docket/containers may fit in current process.
You don't have to, but it would probably be best to keep it consistent. You could run your live server on barebones OS as you normally would, then create a docker image to mirror that. However, as we all know, mirrors often get out of sync fairly quick. But if you use the same image and only update using a Fabric or Ansible script, then you can be pretty certain that all your environments are identical. Not to mention that any newly created environments will brought up to the same point in no time at all.
If you don't want to use Docker in production you could use something like Packer to build Docker base images (http://www.packer.io/docs/builders/docker.html) and VM images (AMIs, OpenStack, whatever) using the same templates.
There would still be some differences, like if the kernel version on your local machine didn't match.
Note: I haven't tried this. Maybe it's a bad idea.
There would still be some differences, like if the kernel version on your local machine didn't match.
Note: I haven't tried this. Maybe it's a bad idea.
if your environment is standardized and stripped down you can run something that is very close to prod in your dev environment.
Sounds a lot like FreeBSD's release process, though personally I find that limiting!
The problem is of course that 'very close to' is not 'equal to'.
My own thoughts in this space are detailed at http://stani.sh/walter/cims/ and http://stani.sh/walter/pfcts/
Sounds a lot like FreeBSD's release process, though personally I find that limiting!
The problem is of course that 'very close to' is not 'equal to'.
My own thoughts in this space are detailed at http://stani.sh/walter/cims/ and http://stani.sh/walter/pfcts/
I'm using Docker as different than most people intended: To grade student programming assignments on different physical machines. And I can distribute the docker images to students, so they stop complaining "But it works on my machine". So it is a different usage of packaging and shipping code.
IMHO every open source project should come with either a Vagrantfile or Dockerfile, if only to allow potential users to quickly try out the project in a known good environment.
Of course then you risk developers of libraries/tools only developing/testing against that particular environment, but that could be solved by having a variety of Docker/Vagrantfiles integrated with a CI system.
Of course then you risk developers of libraries/tools only developing/testing against that particular environment, but that could be solved by having a variety of Docker/Vagrantfiles integrated with a CI system.
That is very fragile, you want to make sure your code is not that dependent on a very specific environment really.
Yes, hence the second sentence: "Of course then you risk developers of libraries/tools only developing/testing against that particular environment, but that could be solved by having a variety of Docker/Vagrantfiles integrated with a CI system."
Sort of - the 'works on my machine' issue is definitely something Docker helps with in DevOps settings too.
I'm just start to play with Docker, it seems like a nice way to build out contained systems that you don't want to change much ... but feels like it could lead to problems when it comes time to do updates to software inside the container. It feels like Docker offloads alot of responsibility for security updates from the ops to people in development. Or am I missing something ...
Docker provides more flexibility. So while you can have developers managing security updates that is definitely not the only way of handling things.
I see a process where ops are responsible for the base images (base os images, and layers for different software stacks) and can handle security updates at will. Developers can maintain small dockerfiles that just install their application on top of the ops-managed image. ops can then be in charge of building (and re-building) the images and deploying them. Of course, that can all be fine-tuned to your liking.
I see a process where ops are responsible for the base images (base os images, and layers for different software stacks) and can handle security updates at will. Developers can maintain small dockerfiles that just install their application on top of the ops-managed image. ops can then be in charge of building (and re-building) the images and deploying them. Of course, that can all be fine-tuned to your liking.
It forces co-operation. As someone doing ops, and with years and years of development: ops teams should not do software upgrades without cooperating with developers and re-testing the full stack anyway. The odds of seemingly "safe" OS upgrades breaking applications are often high.
If used properly Docker enforces pushing those upgrades back up the pipeline as far as they should.
If used properly Docker enforces pushing those upgrades back up the pipeline as far as they should.
That's exactly right. Upgrades should be higher up in the process and involve the development and testing team. It doesn't mean the same people who run production can't also participate and offer guidance in the upgrade process upstream.
When you want to do updates, you can just replace the old container with a new one reflecting the updates.
The conceptual leap that I missed initially is the use of volumes, either mapped to data containers or to directories on the host system. These allow the persistent data used by the application in the container to live on, and for the container itself to be seen as immutable and easily replaceable.
> Docker is operating system level virtualization. Unlike hypervisor virtualization, where virtual machines run on physical hardware via an intermediation layer ("the hypervisor"), containers instead run user space on top of an operating system's kernel. That makes them very lightweight and very fast.
This is one of the best explanations of Docker I've read. (Altho I suppose it requires knowing what Hypervisor is, if just at a topical level...perhaps the same for a "kernel")
This is one of the best explanations of Docker I've read. (Altho I suppose it requires knowing what Hypervisor is, if just at a topical level...perhaps the same for a "kernel")
Except that this also describes vserver, lxc, solaris zones, freebsd jails, and numerous other container virtualization solutions.
Docker is best described as application virtualization.
Docker is best described as application virtualization.
And we're back to square 1. What does application mean in this context? I have a really hard time figuring out what Docker really is.
Docker was a wrapper around lxc which ran an executable as PID1.
Docker is now a wrapper around libcontainer which runs an executable as PID1, with a build system for those containers (Dockerfiles) and support for features exposed by libcontainer (ports, bind mounts, etc).
The use case of "operating system level virtualization" is "I'm already running Linux, Solaris, BSD, AIX, or whatever, but I want to run another copy on top of it in order to segment off my clients/webserver/business unit/whatever". There's an implication that there'll be a real init as PID1, you may want to run sshd or another access point to let users in, and it's a "pet" in the pets v. cattle parlance.
The use case of Docker is "I built my application with this set of libraries on top of Ubuntu, but I want to deploy it alongside applications with different versions of libraries (also on Ubuntu or RHEL or whatever), and my application is the only thing that'll be running".
Remember when you got tarballs to unpack in /opt/${application} which came with start scripts which started with "export LD_LIBRARY_PATH='/opt/oracle/lib'"? Docker does that in a modern way. It can also do more than that with etcd, fleet, haproxy, and bits tacked on top to shuffle containers around, but that's the core of docker.
You, the developer, ship your application along with all the libraries it needs and it runs in a container on top of Ubuntu but it's using RHEL glibc/libwhatever without starting logind and all the other services associated with RHEL.
Docker is now a wrapper around libcontainer which runs an executable as PID1, with a build system for those containers (Dockerfiles) and support for features exposed by libcontainer (ports, bind mounts, etc).
The use case of "operating system level virtualization" is "I'm already running Linux, Solaris, BSD, AIX, or whatever, but I want to run another copy on top of it in order to segment off my clients/webserver/business unit/whatever". There's an implication that there'll be a real init as PID1, you may want to run sshd or another access point to let users in, and it's a "pet" in the pets v. cattle parlance.
The use case of Docker is "I built my application with this set of libraries on top of Ubuntu, but I want to deploy it alongside applications with different versions of libraries (also on Ubuntu or RHEL or whatever), and my application is the only thing that'll be running".
Remember when you got tarballs to unpack in /opt/${application} which came with start scripts which started with "export LD_LIBRARY_PATH='/opt/oracle/lib'"? Docker does that in a modern way. It can also do more than that with etcd, fleet, haproxy, and bits tacked on top to shuffle containers around, but that's the core of docker.
You, the developer, ship your application along with all the libraries it needs and it runs in a container on top of Ubuntu but it's using RHEL glibc/libwhatever without starting logind and all the other services associated with RHEL.
> I built my application with this set of libraries on top of Ubuntu, but I want to deploy it alongside applications with different versions of libraries (also on Ubuntu or RHEL or whatever)
If two servers require different versions of a library, I see that as not a use case but a problem of technical debt, to be solved by repackaging and retesting with the correct version.
> export LD_LIBRARY_PATH='/opt/oracle/lib'
I guess some people are required to run such badly-maintained software, but I don't see why anyone's excited about the prospect.
If two servers require different versions of a library, I see that as not a use case but a problem of technical debt, to be solved by repackaging and retesting with the correct version.
> export LD_LIBRARY_PATH='/opt/oracle/lib'
I guess some people are required to run such badly-maintained software, but I don't see why anyone's excited about the prospect.
You're missing the point, which isn't that "badly-maintained software" and "technical debt" are the problems. It's that I, as a developer who writes Python which runs on Fedora, EL[6|7], Gentoo, Debian, Ubuntu, and Arch have to go through an inordinate amount of testing and hoop-jumping in order to make sure that my application runs on EL6 and Fedora, to say nothing of systems which aren't guaranteed to have the same package names, library locations, or major versions of Python, much less having it at all (especially a problem on EL6 and Gentoo).
It makes sense for me, as a developer, not to go through repackaging and retesting for 7 distros with 4 different packaging paradigms and 3 init systems when I could simply tell you to run a container. None.
Disk space and memory are now cheap. Statically linked binaries are coming back. Containers as the new /opt strike a middle ground between distro portability and developer effort. It must be nice for you to live in an ivory tower.
It makes sense for me, as a developer, not to go through repackaging and retesting for 7 distros with 4 different packaging paradigms and 3 init systems when I could simply tell you to run a container. None.
Disk space and memory are now cheap. Statically linked binaries are coming back. Containers as the new /opt strike a middle ground between distro portability and developer effort. It must be nice for you to live in an ivory tower.
If you're throwing proprietary binaries over the wall for clients who aren't paying enough to support all their platforms, I guess picking one is a necessary evil. But it seems strange to set up a container, which is essentially your own custom distro which nobody else uses, rather than just picking a well-supported distro.
And if I had a goal of minimizing any interaction between my code and its environment, Linux syscalls are a much bigger API than I would choose. It's not as if you can claim to support a distro whose kernel or docker version you haven't tested on.
And if I had a goal of minimizing any interaction between my code and its environment, Linux syscalls are a much bigger API than I would choose. It's not as if you can claim to support a distro whose kernel or docker version you haven't tested on.
chroot + buzzwords (but it does ultimately have a legitimately simpler interface/workflow)
Sort of yes, sort of no. I also agree that Docker is incredibly buzzwordy and that most of the bullshit and hype around Docker could be done with other tools
But the libcontainer networking stuff and integration with cgroups does provide more segmentation than chroots, and the networking parts are nice.
Granted, I think the best possible use case for Docker is in shipping fat apps (ala OSX or Windows) so Spotify for Linux can run on anything that supports Docker instead of anything which supports dpkg, but eh.
But the libcontainer networking stuff and integration with cgroups does provide more segmentation than chroots, and the networking parts are nice.
Granted, I think the best possible use case for Docker is in shipping fat apps (ala OSX or Windows) so Spotify for Linux can run on anything that supports Docker instead of anything which supports dpkg, but eh.
For me, Docker brings to mind the Henry Spencer quote, originally implicating Microsoft, that "Those who don't understand Unix are condemned to reinvent it"[1].
I totally understand the enthusiasm around OS containers. I forget sometimes that this is a new thing on Linux.
Running Solaris and FreeBSD is like living in the future!
[1] The full quote ends "...poorly", but Docker seems to done well-enough, just rather late to the party.
I totally understand the enthusiasm around OS containers. I forget sometimes that this is a new thing on Linux.
Running Solaris and FreeBSD is like living in the future!
[1] The full quote ends "...poorly", but Docker seems to done well-enough, just rather late to the party.
But it isn't a new thing on Linux. lxc, vserver, openvz, and other projects also do what zones and jails do. Docker is not the same.
That's fair, I was writing imprecisely.
The functionality has existed (at least primitively) on Linux for some time, but culturally Linux admins have been more drawn to HW level virtualization instead.
OS-level virtualization has been a part of the FreeBSD and Solaris cultures for much longer.
Another fair argument is that Docker does such a good job at abstracting the configuration and management of OS-level virtualization that it truly changes things.
Maybe. But if that's true, it means that Linux admins have ignored this hugely useful technology because it was hidden behind an impenetrable wall of text-based configuration.
The functionality has existed (at least primitively) on Linux for some time, but culturally Linux admins have been more drawn to HW level virtualization instead.
OS-level virtualization has been a part of the FreeBSD and Solaris cultures for much longer.
Another fair argument is that Docker does such a good job at abstracting the configuration and management of OS-level virtualization that it truly changes things.
Maybe. But if that's true, it means that Linux admins have ignored this hugely useful technology because it was hidden behind an impenetrable wall of text-based configuration.
Docker is great because it allows you to avoid simulating an entire virtual machine.
Think how ridiculous it is that in many cases today we run an entire simulated computer, with a full general-purpose operating system, just to power a single application (e.g. web server, database, etc).
Think how ridiculous it is that in many cases today we run an entire simulated computer, with a full general-purpose operating system, just to power a single application (e.g. web server, database, etc).
Why is it so ridiculous? It's pretty cheap these days.
It's not ridiculous at all, given production deployments need segmentation to maximize security. Docket may be fine for Dev environments with no real customer data being used, but to deploy to production if you use Docker, you will get shot... by the security guys.
I don't know about this, if you are following best practices and then add "with docker" you don't really change anything for the worse. There's just a whole new layer of best practices with docker that you have to learn, namely single-process container ideas and volume/image separation.
If you're using Docker in development and then not using it in production, I hope you're using chef too, or something like it. The point to me is embracing the whole idea of "replaceable, reusable, disposable" components and DRY. If you're repeating by hand any complex steps in production that you also did in your development environment; never mind the time wasted, you now have twice as many chances to make mistakes.
If you're using Docker in development and then not using it in production, I hope you're using chef too, or something like it. The point to me is embracing the whole idea of "replaceable, reusable, disposable" components and DRY. If you're repeating by hand any complex steps in production that you also did in your development environment; never mind the time wasted, you now have twice as many chances to make mistakes.
There are plenty of wasted resources. Virtualization itself incurs a 30% overhead.
Do you have a citation for that? I did a dry run of a presentation for an engineer and her first question was "exactly how much fewer resources"? Would love to have a source for that in case I get asked again.
http://www.altechnative.net/2012/08/04/virtual-performance-p...
http://serverfault.com/questions/261974/how-much-overhead-do...
These are good examples of there is no such thing as "the" virtualization and even front runners have a near 10:1 ratio of performance.
Its like talking about "the" sorting algorithm.
Its especially tough since there is no "the" benchmark. So if you're running a compile farm two years ago, the first article is quite relevant, otherwise maybe not.
I've done a lot personally and professionally with virtualization and as a gut reaction you can usually justify and afford better specs because not all images will max out CPU at the same time, etc etc so in terms of production "stuff" done per hardware dollar you get a little more done with visualization assuming a sensible architecture. Then again you blow more money and labor and labor is money, on software for virtualization. Then again management and backup is usually simpler with virtualization, except when it isn't like when vmware and the NAS kills your mysql instances on an image when you run a backup and it starves the kernel for 60+ seconds of IO whoops.
On average it ends up being about the same performance as non virtualized for a given amount of $$$$ but its much more flexible and responsive to change, which sometimes is worth it.
http://serverfault.com/questions/261974/how-much-overhead-do...
These are good examples of there is no such thing as "the" virtualization and even front runners have a near 10:1 ratio of performance.
Its like talking about "the" sorting algorithm.
Its especially tough since there is no "the" benchmark. So if you're running a compile farm two years ago, the first article is quite relevant, otherwise maybe not.
I've done a lot personally and professionally with virtualization and as a gut reaction you can usually justify and afford better specs because not all images will max out CPU at the same time, etc etc so in terms of production "stuff" done per hardware dollar you get a little more done with visualization assuming a sensible architecture. Then again you blow more money and labor and labor is money, on software for virtualization. Then again management and backup is usually simpler with virtualization, except when it isn't like when vmware and the NAS kills your mysql instances on an image when you run a backup and it starves the kernel for 60+ seconds of IO whoops.
On average it ends up being about the same performance as non virtualized for a given amount of $$$$ but its much more flexible and responsive to change, which sometimes is worth it.
Docker to me makes things an order of magnitude (or two) easier than previously possible. In time these additional layers will be as central as virtualization itself was in arriving with VMware.
After having used many years of virtualization stacks (openvz, xen, vmware, kvm), I believe using docker on production is still quite tricky, if not very tricky.
Especially for larger apps, at the very moment it's like juggling eggshells.
It requires a great deal of orchestration as you gradually involve many docker instances and especially when the host is involved (volumes, links). So yeah , I wouldn't call it actual "containerizing" the app and it's more like containerizing "processes". I for one would love to see a day where openvz gets docker-like cli and having repository for creating/pushing/pulling images.
My favorite quote from the article/interview: "Its mission is basically: make it easy to package and ship code"
Every time I see a post like this, I see lots of folks are super exited, but I still fail to see benefits, perhaps is the nature of my/our architecture/setup.
So, let's say I have a fairly typical setup, 1 web app, 1 database. Normally I would have 2 VM instances to start with. When I need to scale I add more nodes for web and/or db... both are load balanced, of course.
How would docker help me? I guess I should google for docker use cases, but I thought I'll try my luck here too.
Test/dev environments are pretty close to production ones.
It seems there is a lot of hype, but I fail to see it through. I'm wondering if all this hype is artificial, time will tell of course.
Thanks.
So, let's say I have a fairly typical setup, 1 web app, 1 database. Normally I would have 2 VM instances to start with. When I need to scale I add more nodes for web and/or db... both are load balanced, of course.
How would docker help me? I guess I should google for docker use cases, but I thought I'll try my luck here too.
Test/dev environments are pretty close to production ones.
It seems there is a lot of hype, but I fail to see it through. I'm wondering if all this hype is artificial, time will tell of course.
Thanks.
I don't understand the part with scaling with containers on the same host. I mean if it's the same host, what kind of scaling is it?
The scaling part is generally about running and managing multiple containers on multiple machines. In that sense, the advantage of going from virtualization to containerization is the same as going from bare-metal to virtualization: better computation density and more agile deployments.
On the one hand, containers are more efficient than virtual machines. Hence, you get more efficient usage of your hardware. Also, you can pack more containers in a given host than if you use virtualization. Net result: less hardware to run the same services. In turn, this also readuces the network load because the more services in the same machine, the less traffic on your network and better latencies between colocated services.
On the other hand, containers (as envisioned by docker) are much faster to deploy. You build images which are compositions of layers. You have a base layer containing the base OS, and then a "stack" layer containing your application stack, and finally your app on another layer. An image is a consolidated view of all these layers, but the layers can be fetched and managed independently. Hence, you can deploy containers much faster (no need to re-download every layer each time you want to deploy a new service). Additionally, they build faster and they launch faster (nearly as fast as a non-containerized application). This allows you to do even cooler things such as socket-activated containers, which would be painful with the startup times of virtual machines.
Obviously, this blends very well with the idea of building applications in terms of distributed services (or components or whatever). As a result, there are multiple projects pushing for "cluster-aware" (replicated, varying levels of consistency) services. People from the virtualization camp was already pushing for that, but now it is even more important because containers are leaner and cheaper to deploy.
On the one hand, containers are more efficient than virtual machines. Hence, you get more efficient usage of your hardware. Also, you can pack more containers in a given host than if you use virtualization. Net result: less hardware to run the same services. In turn, this also readuces the network load because the more services in the same machine, the less traffic on your network and better latencies between colocated services.
On the other hand, containers (as envisioned by docker) are much faster to deploy. You build images which are compositions of layers. You have a base layer containing the base OS, and then a "stack" layer containing your application stack, and finally your app on another layer. An image is a consolidated view of all these layers, but the layers can be fetched and managed independently. Hence, you can deploy containers much faster (no need to re-download every layer each time you want to deploy a new service). Additionally, they build faster and they launch faster (nearly as fast as a non-containerized application). This allows you to do even cooler things such as socket-activated containers, which would be painful with the startup times of virtual machines.
Obviously, this blends very well with the idea of building applications in terms of distributed services (or components or whatever). As a result, there are multiple projects pushing for "cluster-aware" (replicated, varying levels of consistency) services. People from the virtualization camp was already pushing for that, but now it is even more important because containers are leaner and cheaper to deploy.
Thanks for trying to explain.
In my mind scale means use more resources, but in this case I fail to see benefits, except quicker env setup for the app, but for that there are automation services.
Yes, cluster-aware apps, if I want to scale 1 app, I'll create dedicated nodes/vm instances for it, how are containers helping? Why woudl I need multiple containers on the same host, for each app instance?
Thanks again.
In my mind scale means use more resources, but in this case I fail to see benefits, except quicker env setup for the app, but for that there are automation services.
Yes, cluster-aware apps, if I want to scale 1 app, I'll create dedicated nodes/vm instances for it, how are containers helping? Why woudl I need multiple containers on the same host, for each app instance?
Thanks again.
You are missing part of the story. If scaling is just "using more resources" then I can scale infinitely. I can run "while true" on as many hosts as you want. What I'm trying to point out here is that scaling is about doing more useful work, not just using more resources.
Why would you need more containers on a single host? Because apps are usually not uniform. If you dedicate one host per service, then you need hosts to serve the maximum computation power at the peak consumption required at any point in the app's lifetime. Also, VMs are heavyweight, which means that it is painful to break your app into small, independent services that you can spin up and down quickly. Containers are much better at enabling you to modularize (and thus be able to scale on a finer grain) your application.
Now you may argue that you can spin more AWS instances when under load. But spinning up VMs takes time, much more time than spinning up containers (we're talking on the order of minutes vs milliseconds here). This may seem a small difference, but pair it with socket activation for instance and suddenly you go from having to carefully manage your scaling to getting it quasi-automated for free.
Finally, there's the ecosystem. You can create dedicate nodes/vm instances, sure. But watch a demo of CoreOS's fleet automatically and nearly-instantly spinning up and maintaining coordinated containers [1] and you may get a feeling of why people is so excited about this whole thing.
[1] http://coreos.com/blog/cluster-level-container-orchestration...
Why would you need more containers on a single host? Because apps are usually not uniform. If you dedicate one host per service, then you need hosts to serve the maximum computation power at the peak consumption required at any point in the app's lifetime. Also, VMs are heavyweight, which means that it is painful to break your app into small, independent services that you can spin up and down quickly. Containers are much better at enabling you to modularize (and thus be able to scale on a finer grain) your application.
Now you may argue that you can spin more AWS instances when under load. But spinning up VMs takes time, much more time than spinning up containers (we're talking on the order of minutes vs milliseconds here). This may seem a small difference, but pair it with socket activation for instance and suddenly you go from having to carefully manage your scaling to getting it quasi-automated for free.
Finally, there's the ecosystem. You can create dedicate nodes/vm instances, sure. But watch a demo of CoreOS's fleet automatically and nearly-instantly spinning up and maintaining coordinated containers [1] and you may get a feeling of why people is so excited about this whole thing.
[1] http://coreos.com/blog/cluster-level-container-orchestration...
Yeah, I have some guesses but if anyone here can clear that up, I'd appreciate it.
Because Docker just makes more sense. Why should I run multiple versions of the whole OS for near-to-identical setups? That is just a waste of resources. Docker makes much more sense when you have multiple applications you want in separate instances on the same stack. There benefits far outweigh the disadvantages in my experience.
Yes, I can see a great deal of benefits for development part, _if_ you're sticking with the same OS. Currently I have a bunch VMs (using kvm/qemu + libvirt) that are created and dropped as needed. The biggest advantages I see is that I can create a near identical environment with our VPSs... + I can test different OSs too (Linuxe vs *BSD).
My current pain point is the hdd space, since I have to resize VMs storage when needed. Even though it was automated, but still I'd prefer to use some shared hdd space and not to worry about it.
My current pain point is the hdd space, since I have to resize VMs storage when needed. Even though it was automated, but still I'd prefer to use some shared hdd space and not to worry about it.
Containers are the new craze, because they're much lighter-weight and easier to deploy than full-system virtual machines. Docker is simply a convenient approach to container distribution and deployment, designed to package up an application as a self-contained invokable container.
Excuse me, but isn't this essentially chroots reinvented?
That is exactly it, with a couple more isolation features added. So more than just the FS but also some memory / cpu limiting and general sandboxing.
Interestingly most "real virtualization" requires somewhat recent hardware but LXC containers don't take much if any, so "free machine off junk heap" is good enough for experimentation.
Historically there have been strange intercontainer isolation problems so don't assume its as perfect as hardware assisted virtualization.
Also you're sharing a kernel which is both good and bad, if you were hoping to test out a new kernel or run an entirely different OS, thats too bad.
Edited to add, another fun analogy if you like the chroot analogy, is when spinning up a process was glacial on windows (but not too bad on linux) that lead to the development and push to use threads which were a huge win on slow windows not so much on linux but we got dragged along anyway. In a similar manner, full virtualization is really slow to deploy and spin up and spin down, containers are smaller and weaker but much like threads vs processes are much faster to start up or shutdown. A really low latency or fast spinup spindown virtualization tech would probably wipe most demand for containerization. Or what I'm getting at is, spawning processes on windows was super slow, so we got threads, and spawning full hardware virt is slow, so we get containerization aka chroot++
Interestingly most "real virtualization" requires somewhat recent hardware but LXC containers don't take much if any, so "free machine off junk heap" is good enough for experimentation.
Historically there have been strange intercontainer isolation problems so don't assume its as perfect as hardware assisted virtualization.
Also you're sharing a kernel which is both good and bad, if you were hoping to test out a new kernel or run an entirely different OS, thats too bad.
Edited to add, another fun analogy if you like the chroot analogy, is when spinning up a process was glacial on windows (but not too bad on linux) that lead to the development and push to use threads which were a huge win on slow windows not so much on linux but we got dragged along anyway. In a similar manner, full virtualization is really slow to deploy and spin up and spin down, containers are smaller and weaker but much like threads vs processes are much faster to start up or shutdown. A really low latency or fast spinup spindown virtualization tech would probably wipe most demand for containerization. Or what I'm getting at is, spawning processes on windows was super slow, so we got threads, and spawning full hardware virt is slow, so we get containerization aka chroot++
Alright, thanks a lot for the explanation then! Then my concept wasn't entirely wrong.
It's nice to have logically independent code bases on separate environments just so that they don't step over each other toes with dependencies etc. and can be moved around easily.
This is it exactly. When I can develop locally using my favorite editor and my favorite OS, but then deploy -- still locally, mind -- to what is more or less the production environment... This has huge benefits for avoiding the "works on my box" problems that arise as code makes its way to prod.