Faster Web Development With Virtual Machines at deviantART(dt.deviantart.com)
dt.deviantart.com
Faster Web Development With Virtual Machines at deviantART
http://dt.deviantart.com/blog/35461946/
15 comments
I started using vagrant last week to debug my Chef scripts and it is indeed awesome. Vagrant uses Chef (chef-solo) configure your VM from bare image to working server.
I see that is has a plug in system for other provision systems like Puppet, but the chef support works great out of the box.
I was hoping to contribute to the project by integrating fog the ruby library which talks to many cloud systems including EC2 so someone could build an image via Chef or Puppet, dev and test in their image, then provision and upload to the cloud. Could probably do this without integrating with vagrant by building a clever rake or capistrano script.
I see that is has a plug in system for other provision systems like Puppet, but the chef support works great out of the box.
I was hoping to contribute to the project by integrating fog the ruby library which talks to many cloud systems including EC2 so someone could build an image via Chef or Puppet, dev and test in their image, then provision and upload to the cloud. Could probably do this without integrating with vagrant by building a clever rake or capistrano script.
I love vagrant!
It was super easy to create an archlinux base box and i've been using chef to automatically provision environments. Setting up the environment for a new project takes minutes and is both totally automated and separated from my actual OS.
Thanks!
After talking with a few small PHP teams this seems like the logical conclusion whenever growing a team of PHP developers. Its just very hard to perfectly recreate your server environment locally on multiple OS's.
Its a great solution, but I much prefer working with RoR or Python/Django where recreating the server environment locally is much easier.
Its a great solution, but I much prefer working with RoR or Python/Django where recreating the server environment locally is much easier.
How is creating the local server environment on RoR or Python/Django easier than PHP/anything?
I've been developing locally for years and while I can't mimic the exact hardware of the live server, I wouldn't be able to do that regardless of stack I chose.
I've been developing locally for years and while I can't mimic the exact hardware of the live server, I wouldn't be able to do that regardless of stack I chose.
As Pewpewarrows said, you can use pip with a virtualenv to deploy with one command. If your settings.py is any good, you can load your data dump in an sqlite database in one more command, and you're ready to go.
We used to develop historious like this, until we switched to redis/celery. Then it became too much hassle to install these in every machine, plus we wanted to test on the db we used (postgres) rather than sqlite, so we made a VM and now everything's dandy.
We used to develop historious like this, until we switched to redis/celery. Then it became too much hassle to install these in every machine, plus we wanted to test on the db we used (postgres) rather than sqlite, so we made a VM and now everything's dandy.
For the Python world, virtualenvs and pip requirement files make deploying a local copy of a project take about 30 seconds, and it's completely independent of your overall system and all other projects you happen to be working on.
Gotcha. With git and a Phing build script, I can do the same in PHP :)
Bundler was supposed to remedy dependency hell problems and I believe it eventually managed to do that. But boy can I tell you stories about migrating from pre-1.0 bundler versions.
RVM is just great and simple.
What I’m trying to say though: Having done quite some side projects with node, the ruby stuff suddenly feels very enterprisey to me. More and more conventions might make it harder to keep track of everything in the future.
RVM is just great and simple.
What I’m trying to say though: Having done quite some side projects with node, the ruby stuff suddenly feels very enterprisey to me. More and more conventions might make it harder to keep track of everything in the future.
Your last sentence is an example of when I would like to vote "my displeasure" with a down arrow.
As others have mentioned, you make a bold claim, yet fail to give anything concrete as proof that it is "much easier".
As others have mentioned, you make a bold claim, yet fail to give anything concrete as proof that it is "much easier".
even for RoR or python - might make sense for those working on multiple projects simultaneously, using different generations of frameworks
There's RVM in the Ruby world. Anything similar for Python?
virtualenv
I'd say that seems to cover both bases for not needing a VM.
Every language should have something as simple as these. Heck, it should be part of the OS, IMO; why I can't (easily) run two different versions of Firefox with two different profiles at the same time is beyond comprehension.
Every language should have something as simple as these. Heck, it should be part of the OS, IMO; why I can't (easily) run two different versions of Firefox with two different profiles at the same time is beyond comprehension.
It's been a while since I've done this; but:
* Install your different Firefox versions in different locations (so one doesn't overwrite the other)
* Use "firefox -ProfileManager" to create at least one different 'profile', which is Firefox's user data set
* Use "firefox -P <profilename> -no-remote" to launch Firefox. -no-remote allows multiple Firefox instances to run on the same machine. -P profilename is necessary because you need a separate profile for each running instance.
More info at http://kb.mozillazine.org/Command_line_arguments
* Install your different Firefox versions in different locations (so one doesn't overwrite the other)
* Use "firefox -ProfileManager" to create at least one different 'profile', which is Firefox's user data set
* Use "firefox -P <profilename> -no-remote" to launch Firefox. -no-remote allows multiple Firefox instances to run on the same machine. -P profilename is necessary because you need a separate profile for each running instance.
More info at http://kb.mozillazine.org/Command_line_arguments
Ooh, I'll have to try that. I've been using different profiles, but didn't know about the last step.
Still; mindboggling. Different version = different binary launched, which accesses a different and non-locked profile. Why the hell does it instead open a new window of the currently-running version?
As a heads-up to anyone considering this: Sync doesn't (currently) like multiple versions. 3.6.10 won't sync now that I've hooked up some 4 betas.
Still; mindboggling. Different version = different binary launched, which accesses a different and non-locked profile. Why the hell does it instead open a new window of the currently-running version?
As a heads-up to anyone considering this: Sync doesn't (currently) like multiple versions. 3.6.10 won't sync now that I've hooked up some 4 betas.
You do need a VM if you use memcached/redis/some other thing and don't want to install it on your machine. Other than that (for 90% of cases), virtualenv has you covered.
A VM is a constant time investment, doing it for every language is linear with the number of languages.
Unless there's some serious increases in clock speeds, I'm not sure doing it for every language is worth it. As it is, VMs are a great, general purpose system that are easy to set up, use, and replicate.
Unless there's some serious increases in clock speeds, I'm not sure doing it for every language is worth it. As it is, VMs are a great, general purpose system that are easy to set up, use, and replicate.
I've found that developing against a VM is a great tool for a dev. For OpenStack swift, we develop all of our code against a VM. Since we deploy on debian and I develop on a Mac, I can use OS X tools to develop (TextMate, etc) and code against an environment similar to our production. After code works on a VM, we can move to a lab or a staging environment to do more scale testing before deploying to production.
Developing against a VM is not something I did before working at Rackspace, and I find the practice so useful that I will do what I can to continue doing it no matter where I work in the future.
Developing against a VM is not something I did before working at Rackspace, and I find the practice so useful that I will do what I can to continue doing it no matter where I work in the future.
This all VM thing is fine and dandy if you want to have an accessible mirror of your staging machine and reduce unnecessary stages. Although that in itself looks fishy - if developers want to stage something just to see if it works, then something is seriously wrong with your process or developers.
I'm amazed at the level of sophistication in this setup - using virtual machines and TCP proxies that do text replaces seems too much. They could have just built the configurability in the damn application. I have a .NET app that normally gets deployed on three servers and uses various remote resources like Amazon SQS queues, but I easily built a configuration abstraction similar to Rails' environments that lets me deploy on a single machine and even test locally with a debugger attached.
My point is that this problem looks as if it got solved at the wrong level of abstraction. Which, obviously, made it unnecessarily hard and complex. If I had to tackle it, I'd make sure developers do the right thing and try to make the app configurable enough first.
I'm amazed at the level of sophistication in this setup - using virtual machines and TCP proxies that do text replaces seems too much. They could have just built the configurability in the damn application. I have a .NET app that normally gets deployed on three servers and uses various remote resources like Amazon SQS queues, but I easily built a configuration abstraction similar to Rails' environments that lets me deploy on a single machine and even test locally with a debugger attached.
My point is that this problem looks as if it got solved at the wrong level of abstraction. Which, obviously, made it unnecessarily hard and complex. If I had to tackle it, I'd make sure developers do the right thing and try to make the app configurable enough first.
The mind boggles. There's probably a rockstar in there looking to put something on their CV/resume.
Whatever the case, it smells wrong.
I only skimmed the article but I assume it’s about the benefits of running VMs to have solid dev environments with zero setup cost.
Let me ask another question: Will it become feasible to move code, tools and everything else into web services? Imagine you have your favorite editor (emacs, TM, vim) running in a browser window. Only that it’s live-updating like, e.g., Ethercodes or Wave. And you have very straight-forward access to SCM features like staging hunks of code. Also the code deploys straight to heroku or whatever. It could, of course, also go through closure or sproutcore in case of a fancy js client app.
Do you think it’ll all evaporate (I refuse to say cloud :) eventually?
The thing is I’m really going through a lot of pain setting up my system and other’s systems over and over again. Yes, there are ways to automate that but the effort is too high for most or we would all be doing it right now.
Let me ask another question: Will it become feasible to move code, tools and everything else into web services? Imagine you have your favorite editor (emacs, TM, vim) running in a browser window. Only that it’s live-updating like, e.g., Ethercodes or Wave. And you have very straight-forward access to SCM features like staging hunks of code. Also the code deploys straight to heroku or whatever. It could, of course, also go through closure or sproutcore in case of a fancy js client app.
Do you think it’ll all evaporate (I refuse to say cloud :) eventually?
The thing is I’m really going through a lot of pain setting up my system and other’s systems over and over again. Yes, there are ways to automate that but the effort is too high for most or we would all be doing it right now.
I'm sure it will happen one day, but it's quite a big challenge and you're up against all those bad habits build over the last 20 years. For example, developers really like customization. It's quite hard to provide a hosted environment that is fully customizable.
Also, parenthesis mismatch in paragraph 3.
Also, parenthesis mismatch in paragraph 3.
I remember Heroku had something like this a few years ago. It was pretty easy to get a taste of Rails.
It's not clear to me why they don't just run development on their local machines. I'm not familiar with PHP, though. I'm all for having a build/test vm, though.
Any moderately complex app has a few moving parts. At least you have the web server and the DB. The DB has settings like the default character set, table types etc. These have to be kept in sync across all instances, dev, stage, and prod. The web server also has modules and a conf file with mod_rewrite rules which change and should be kept in sync. Sometimes you update a library or a version of ruby. Add a noSQL server, message bus, file storage, email, etc and you can see how it becomes a serious issue. And the problem grows exponentially the more developers there are each with one or more dev instances which have to get the latest changes and be kept up to date.
These things all have to be dealt with and should not be thrown over the wall and "let the sysadmin deal with it." That's where the value of the OS and infrastructure is a set of code and config files which are kept in source control next to our code and versioned with it is so valuable. Run "vagrant up" and a fresh consistant build every time. Then develop in php, python, perl, ruby or brainfuck if you want.
These things all have to be dealt with and should not be thrown over the wall and "let the sysadmin deal with it." That's where the value of the OS and infrastructure is a set of code and config files which are kept in source control next to our code and versioned with it is so valuable. Run "vagrant up" and a fresh consistant build every time. Then develop in php, python, perl, ruby or brainfuck if you want.
I've had a setup like this going for years. I'm a big fan.
It's unlikely that your server is the same os as your desktop. So you waste time setting up versions of libraries and services for e.g. windows that don't always work quite the same as in production. Plus you can also take advantage of snapshots to mess around with your local vm, confident that you can always roll back to a known version. You can do simplified load testing without actually crashing your whole machine.
It also means your devs can use whatever OS they want - the hard work of setting up a dev environment is already done.
It's unlikely that your server is the same os as your desktop. So you waste time setting up versions of libraries and services for e.g. windows that don't always work quite the same as in production. Plus you can also take advantage of snapshots to mess around with your local vm, confident that you can always roll back to a known version. You can do simplified load testing without actually crashing your whole machine.
It also means your devs can use whatever OS they want - the hard work of setting up a dev environment is already done.
I good PHP environment that models the server is very hard to get going locally. At least on OS X. The only way I could get a good environment for Nginx+MySQL+PHP was to locally run a debian VM in VirtualBox.
Now this doesn't even take into account things like memcached that require a lot more customization.
Now this doesn't even take into account things like memcached that require a lot more customization.
Im confused, this post of yours seems a tad contradictory to your post about "setting up environments in RoR etc".
Developers aren't server administrators (usually). It's not the best use of their time.
I wish I had thought of this when we ran into issues with Windows people and Ruby. Too much of their time was spent fussing with stuff and crying about it. :)
I wish I had thought of this when we ran into issues with Windows people and Ruby. Too much of their time was spent fussing with stuff and crying about it. :)
JRuby cn also be a good way to run ruby on windows.
“So now it should work, right?”
-“I’m getting an error”
“Which error?”
-“Unnamed method in [...]”
Then, a mountain of pain and torture later.
"But now it works, ok?”
-“Ok.”
---
Ever had to fix someone else’s system via Skype or phone? I personally believe writing a rails clone in brainfuck is more fun.
-“I’m getting an error”
“Which error?”
-“Unnamed method in [...]”
Then, a mountain of pain and torture later.
"But now it works, ok?”
-“Ok.”
---
Ever had to fix someone else’s system via Skype or phone? I personally believe writing a rails clone in brainfuck is more fun.
It's easier to give a new developer a VM image than have them set up Apache, 2 Apache modules, PHP, 3 PHP extensions, MySQL, memcached, ImageMagick, mplayer, a queueing system, a custom database daemon for statistics, and a Lucene-based search daemon. And all with the same options as production.
I do acknowledge that this is something that looks very "oh, how obvious!" to the modern crowd, but deviantART is a 10 year old PHP app...
Of course, even if you're running a modern web framework in Ruby/Python/somethingtrendy, a VM makes a lot of sense. It lets you run the exact versions of everything that your production environment is, without having to worry about it.
Of course, even if you're running a modern web framework in Ruby/Python/somethingtrendy, a VM makes a lot of sense. It lets you run the exact versions of everything that your production environment is, without having to worry about it.
Developing in the exact same environment as you deploy in should be standard practice.
This reminds me of facebook development, where you need to have a live fqdn for all of your environments. We used to use a staging server and had a similar problem with huge amounts of small unnatural commits for things you would normally test locally. I ended up moving to using ssh tunnels to a server, where each dev has a subdomain redirecting to their dev box through the tunnel. This has the added benefit that you can work locally from anywhere you have wifi. Whether you use tunnels or VMs, both are a solid way to do development.
If commits were an issue before this, they should really have considered using git or something. You shouldn't aim for fewer commits, but less expensive commits, i.e. nothing goes over a network. Plus you can squash later if you really need to.
Anyway, VMs are cool but has anyone tried developing with NixOS? It seems even more suited for the tasks they want since it has rollbacks and so on: http://nixos.org/
Anyway, VMs are cool but has anyone tried developing with NixOS? It seems even more suited for the tasks they want since it has rollbacks and so on: http://nixos.org/
Isn't the expense of frequent commits the cost of composing an intelligible commit message?
Great Article. As QA Analyst and Engineer, I know first hand about the pains of replicating production environment. Thanks for submission.
I do most of my work in python/django so virtualenv essentially solves much of these problems.
The problem I am running into now is that our production environment is starting to get complicated enough(mysql, redis, 2 different django projects) that testing all the components working together gets more and more difficult all the time.
The problem I am running into now is that our production environment is starting to get complicated enough(mysql, redis, 2 different django projects) that testing all the components working together gets more and more difficult all the time.
Start scripting your environment, the more repeatable your deploy procedure is the better your deploys will be. The easier your deploys are the more frequently you do them, the more frequently you do them, the better your deploys will be. Rinse wash repeat.
Once you have your env scripted with EC2 you can setup / teardown environments easily for testing.
Once you have your env scripted with EC2 you can setup / teardown environments easily for testing.
Having a VM you can trivially reimage is very helpful because you can test that (re)deploying packages to it is going to work, you didn't miss any dependencies that happen to be present on your desktop but not on your production servers, and uninstall won't leave cruft behind.
Another benefit of this approach is that it takes seconds to revert to a good state if you destroyed your dataset in a mishap. use snapshots or just restart VM from original image. all while using your preferred code editor (the app source is mounted from the host OS)
I wonder how long this took to set up? And why they don't run their production servers as VMs as well?
Edit: I think it's an awesome idea, btw, and something we try to push to web service shops at GridCentric.
Edit: I think it's an awesome idea, btw, and something we try to push to web service shops at GridCentric.
Only use VMs or sandboxes on production-like machines.
Local development doesn't scale.
Local development doesn't scale.
If anyone is looking for an easy, low-risk, documented method of beginning development in VMs, check out Vagrant[1], which is what I wrote this for. Creating dev VMs from scratch is pretty tedious and there are a lot of steps involved. Vagrant abstracts this away and provides a tool to automate all of this in a reproducible manner.
[1]: http://vagrantup.com