Using Docker as a Python Development Environment(continuousdelivery.uglyduckling.nl)
continuousdelivery.uglyduckling.nl
Using Docker as a Python Development Environment
http://continuousdelivery.uglyduckling.nl/uncategorized/using-docker-as-a-python-development-environment/
6 comments
I'm confused how they created the docker image with python, postgresql and apache in the first place? Is it the FROM python_webapp that defines that? python_webapp being the name of a docker image already defined? How hard is it to set up the docker image initially?
Edit: NVM, it is documented here (I missed it the first read through) http://continuousdelivery.uglyduckling.nl/continuous-deliver...
Edit: NVM, it is documented here (I missed it the first read through) http://continuousdelivery.uglyduckling.nl/continuous-deliver...
I'm more curious as to how to get a lightweight Docker container running Python. The one they use pulls in an Ubuntu Quantal image, which isn't something that I'd normally use (I'm more in the RHEL/CentOS camp).
So, how would one go about building something like this that isn't tied to a container that pulls in a whole OS? Or is that just not the way these things work? Or am I just too used to virtualenv?
So, how would one go about building something like this that isn't tied to a container that pulls in a whole OS? Or is that just not the way these things work? Or am I just too used to virtualenv?
You can use CentOS if you prefer that. Also the OS is pulled just once.
to be honest i still usually implement virtualenv when constructing a container.
This way I'm sure that I'm not interfering with the os-level python dependencies, and any activation of the environment is handled by a "python-webapp" baseimage that I use for all of my projects.
This way I'm sure that I'm not interfering with the os-level python dependencies, and any activation of the environment is handled by a "python-webapp" baseimage that I use for all of my projects.
I'm not sure about this guy's list of issues with virtualenv.
Point 1 (he forgets to activate it) is not an issue at all. You could forget to "activate" your docker too.
Point 3 (he doesn't want to run it in production) but running docker in production is okay?
That said, he made a good point about non-python dependencies.
Point 1 (he forgets to activate it) is not an issue at all. You could forget to "activate" your docker too.
Point 3 (he doesn't want to run it in production) but running docker in production is okay?
That said, he made a good point about non-python dependencies.
Regarding point 1, the way in which I start the server has fundamentally changed but is still only one command - I know I have to do "docker run" to start it, rather than "python run.py" or similar. In virtualenv I have to remember two commands; "workon bla" and "python run.py".
Regarding point 3, yes I think docker in production is ok, or soon will be :)
Regarding point 3, yes I think docker in production is ok, or soon will be :)
How about you use a simple virtualenv wrapper such as https://pypi.python.org/pypi/vex ?
Then you can easily type 'vex myenv python myapp.py', no need to spin up a linux container for simple development.
Then you can easily type 'vex myenv python myapp.py', no need to spin up a linux container for simple development.
> no need to spin up a linux container for simple development
But why not do this, anyway?
These containers are very cheap and come with very little overhead, they are truly isolated and they are easily shareable. I'd say there is no need not to spin up a container for everything, it's not like you're going to run out of RAM...
But why not do this, anyway?
These containers are very cheap and come with very little overhead, they are truly isolated and they are easily shareable. I'd say there is no need not to spin up a container for everything, it's not like you're going to run out of RAM...
While they do make sense in certain situations, they also add more complications. On linux, it's much easier, but If I want to run this on windows, or osx, they tell you to run Docker inside of a full VBox VM (and maintain it), this is not ideal as VBox destroys my battery and other annoyances.
You also have to maintain those containers/images (not to mention lug around a 600+mb base OS image and update it). I am not sure how Docker handles keeping images updated, I assume overlayfs makes it easy to keep your base docker OS images updated, but not sure how it'd handle certain package configurations.
You also have to maintain those containers/images (not to mention lug around a 600+mb base OS image and update it). I am not sure how Docker handles keeping images updated, I assume overlayfs makes it easy to keep your base docker OS images updated, but not sure how it'd handle certain package configurations.
To be fair, python + anything that needs compiling on windows adds complications.
If I had a bunch of money to invest I pay to:
- Get non-virtualenv libraries moved to virtualenv.
- Get non virtualenv libraries working in windows.
If I had a bunch of money to invest I pay to:
- Get non-virtualenv libraries moved to virtualenv.
- Get non virtualenv libraries working in windows.
> If I want to run this on windows, or osx
Yeah, it's completely nonsense then, the whole point for this is to be extremely lightweight; if you have to run a full VM you may just as well use it directly.
> You also have to maintain those containers/images
Well, that's true, but due to the containers being lightweight and focused you don't really have that much to maintain. I'm not a supporter of one-process-per-container philosophy, but even 3 processes/services inside a container are easier to manage than a full VM.
> not to mention lug around a 600+mb base OS image and update it
Last I checked the base Ubuntu image was ~120Mb.
> I am not sure how Docker handles keeping images updated
You just rebuild them. Thanks to Dockerfiles it's fully automated, so you can plug it into your CI.
The whole point of Docker containers is that they are very easy and cheap to build, run, share and destroy. Personally, I use Docker as a replacement for VirtualBox - my containers are nearly-full-OSes, with sshd, cron, syslog etc. running on them. But building a new one for a new project - if the project doesn't need something exotic like different distro - takes tens of seconds at most and starting/running a new instance (which is equivalent of starting VirtualBox image) takes a few seconds at most.
The important thing to remember about Docker is that you can have as little or as much of an OS inside of a container as you want. The only thing shared between the container and native OS is the kernel - inside the container you can install any distro you want or even get by without any explicit distro (although I don't know if someone tried this yet). You can have a full-fledged OS if you really need it - you can run the container, ssh on it and configure everything you need to. Or you can just run a single process inside a container. Or you can stay in the middle. Whatever you do, Docker gracefully adapts, unlike a VM, which will always need heaps of RAM, have problems with interop and so on.
Over in *BSD land they have jails - I've been using them for years before Docker existed - and they are really proven piece of technology. One interesting use case for them is in PC-BSD, which is FreeBSD-derived desktop system. There's this nifty "wizard-like" GUI for spinning up jails for installing packages from outside the official package manager (like from sources or ports). The system takes care of tunneling, symlinking userland and so on; you can install/build/compile three different versions of Firefox in their respective jails and run them seamlessly alongside each other on your native desktop. It's just one of the possibilities this technology - I mean very cheap kernel-level virtualization - can be used for.
Yeah, it's completely nonsense then, the whole point for this is to be extremely lightweight; if you have to run a full VM you may just as well use it directly.
> You also have to maintain those containers/images
Well, that's true, but due to the containers being lightweight and focused you don't really have that much to maintain. I'm not a supporter of one-process-per-container philosophy, but even 3 processes/services inside a container are easier to manage than a full VM.
> not to mention lug around a 600+mb base OS image and update it
Last I checked the base Ubuntu image was ~120Mb.
> I am not sure how Docker handles keeping images updated
You just rebuild them. Thanks to Dockerfiles it's fully automated, so you can plug it into your CI.
The whole point of Docker containers is that they are very easy and cheap to build, run, share and destroy. Personally, I use Docker as a replacement for VirtualBox - my containers are nearly-full-OSes, with sshd, cron, syslog etc. running on them. But building a new one for a new project - if the project doesn't need something exotic like different distro - takes tens of seconds at most and starting/running a new instance (which is equivalent of starting VirtualBox image) takes a few seconds at most.
The important thing to remember about Docker is that you can have as little or as much of an OS inside of a container as you want. The only thing shared between the container and native OS is the kernel - inside the container you can install any distro you want or even get by without any explicit distro (although I don't know if someone tried this yet). You can have a full-fledged OS if you really need it - you can run the container, ssh on it and configure everything you need to. Or you can just run a single process inside a container. Or you can stay in the middle. Whatever you do, Docker gracefully adapts, unlike a VM, which will always need heaps of RAM, have problems with interop and so on.
Over in *BSD land they have jails - I've been using them for years before Docker existed - and they are really proven piece of technology. One interesting use case for them is in PC-BSD, which is FreeBSD-derived desktop system. There's this nifty "wizard-like" GUI for spinning up jails for installing packages from outside the official package manager (like from sources or ports). The system takes care of tunneling, symlinking userland and so on; you can install/build/compile three different versions of Firefox in their respective jails and run them seamlessly alongside each other on your native desktop. It's just one of the possibilities this technology - I mean very cheap kernel-level virtualization - can be used for.
Don't get me wrong, I've been using containers for many many years. I run pure LXC on my desktop at home for simple containers. Containers are great in many situations. It's a shame LXC didn't get as much hype as Docker (not sure if Docker is LXC based anymore, but was for quite sometime).
However, you do require Docker (or LXC), and you need proper cgroup support in your kernel if you want true isolation. This is perfectly fine for myself and perhaps a few of my developers running Linux, but it starts to look less appealing in other environments.
Personally, I have to maintain multiple FreeBSD servers and even have a local FreeBSD machine for related purposes. I've also had to develop and maintain python applications for SmartOS (solaris based) machines.
Perhaps one day Docker will support Solaris Zones or FreeBSD jails, who knows. :)
However, you do require Docker (or LXC), and you need proper cgroup support in your kernel if you want true isolation. This is perfectly fine for myself and perhaps a few of my developers running Linux, but it starts to look less appealing in other environments.
Personally, I have to maintain multiple FreeBSD servers and even have a local FreeBSD machine for related purposes. I've also had to develop and maintain python applications for SmartOS (solaris based) machines.
Perhaps one day Docker will support Solaris Zones or FreeBSD jails, who knows. :)
>But why not do this, anyway?
It's one more thing that can break, and if it does, it will take time & effort to fix it.
There's also no benefit as far as I can tell. Virtualenv + apt-get solve the environment issue for me...
It's one more thing that can break, and if it does, it will take time & effort to fix it.
There's also no benefit as far as I can tell. Virtualenv + apt-get solve the environment issue for me...
I'm happy that it works for you, but many people have complex infrastructures that aren't easily deployed with virtualenv+apt-get alone.
I wish I could deploy our python apps with virtualenv alone, but before docker we ended up having to create our own deployment system to wrap the virtualenv with all the associated dependencies so it could be pushed to various environments in a deterministic manner.
I wish I could deploy our python apps with virtualenv alone, but before docker we ended up having to create our own deployment system to wrap the virtualenv with all the associated dependencies so it could be pushed to various environments in a deterministic manner.
>I'm happy that it works for you, but many people have complex infrastructures that aren't easily deployed with virtualenv+apt-get alone.
For your development environment? Seriously?
>I wish I could deploy our python apps with virtualenv alone
I thought this was about dev environments? For deployment (and testing deployment) I use a virtual machine too - vagrant/virtualbox + ansible. Yes, I wouldn't want to pollute my computer with that stuff either.
I wouldn't want to develop inside a virtual machine, though.
For your development environment? Seriously?
>I wish I could deploy our python apps with virtualenv alone
I thought this was about dev environments? For deployment (and testing deployment) I use a virtual machine too - vagrant/virtualbox + ansible. Yes, I wouldn't want to pollute my computer with that stuff either.
I wouldn't want to develop inside a virtual machine, though.
A previous python project I worked on had multiple independently developed services, sometimes with conflicting requirements. Production often separated these, but keeping staging and local development environments happy was a constant struggle.
I also like to make sure the devs are working with an environment that translates directly to production (docker makes this much easier now). I really don't enjoy sorting out a blob of code that only works with your specific local configuration:
- ok requires opencv (or other complicated dependency). No problem, that's already in production. - Oh it only builds against the 2.4.7.1 version you have on your machine. - Hmm, 2.4.7.1 has a bug when used with common_lib version X on distro Y - etc.
I also like to make sure the devs are working with an environment that translates directly to production (docker makes this much easier now). I really don't enjoy sorting out a blob of code that only works with your specific local configuration:
- ok requires opencv (or other complicated dependency). No problem, that's already in production. - Oh it only builds against the 2.4.7.1 version you have on your machine. - Hmm, 2.4.7.1 has a bug when used with common_lib version X on distro Y - etc.
If it breaks it's essentially a bug in the kernel.
If you don't see benefits of virtualization then I guess there's no convincing you, and maybe you just really don't need them. But there are people who would benefit from virtualization but don't do it because of the costs of maintaining and running a full VM, and Docker can be a good solution for them.
If you don't see benefits of virtualization then I guess there's no convincing you, and maybe you just really don't need them. But there are people who would benefit from virtualization but don't do it because of the costs of maintaining and running a full VM, and Docker can be a good solution for them.
Python has virtual environments already. I don't see any need to host it in a guest VM.
Because it introduces a lot of complexity with literally zero need or benefit.
how about one command...
$ /path/to/virtualenv/bin/python run.py
$ /path/to/virtualenv/bin/python run.py
>That said, he made a good point about non-python dependencies.
Does he? I have about 5, all of which I can install with one line with apt-get. Only a few libraries have external dependencies anyway.
I could understand for projects/frameworks that require extensive specialized configuration/package installations that might interfere with other things on your system. You don't really have that problem with virtualenv, though. It's 90% isolated on it's own.
Does he? I have about 5, all of which I can install with one line with apt-get. Only a few libraries have external dependencies anyway.
I could understand for projects/frameworks that require extensive specialized configuration/package installations that might interfere with other things on your system. You don't really have that problem with virtualenv, though. It's 90% isolated on it's own.
I have developed in Python for the past 4 years, give or take. Virtualenvs have, hands down, been one of the worst parts of the experience, without exception, and I can not recommend them.
(No, I won't write up an essay about why on HN. If you really want me to, shoot me an email and I'll write an essay about them).
(No, I won't write up an essay about why on HN. If you really want me to, shoot me an email and I'll write an essay about them).
I've been doing this for a while longer (Python heavily for the past 7-ish years) and I've had a great experience working with virtualenvs...
So, I'd love to see that essay :)
So, I'd love to see that essay :)
This could be really useful for non-python webapps.
Gtk and other libraries for stuff like Sound haven't caught up to the virtualenv age + getting them to work can involve ugly hacks.
Gtk and other libraries for stuff like Sound haven't caught up to the virtualenv age + getting them to work can involve ugly hacks.
Why, why should I do this?) What's wrong with python3 and Postgresql which are packaged in last Ubuntu?
How about the principle of not adding unnecessary entities/complications?
How about the principle of not adding unnecessary entities/complications?
It's probably not needed when working on a single project - although even then Docker has some advantages, like being able to have another distro in the image than you have natively - but it makes your life infinitely simpler when you need to work with two or more projects with differing dependencies.
That seems orthogonal to using virtual environments or not. Are you talking about putting them on your dev machine? In that case: reproducible development environments.
"reproducible development environments" - what does this mean?)
I have put pretty complex setup at work on docker and it made things really simple for people who wants to set it up on their work laptop. Especially the new hires (oh and interns).
Of course Docker is more stuff to learn but when it's just a bunch of instructions to follow on a readme file they would just run those and have the system running (mostly without much trouble) and later figure out what this Docker thing really is, if they are in to it.