Virtualization: good idea for startups?
24 comments
Phil Pearson here from Broadband Mechanics. We use Xen for all our hosting... the killer feature for us is the performance isolation. If one site gets too busy for its VM, it might go down, but it doesn't hurt the other sites on the same physical server nearly as much as without virtualization.
The ability to (relatively) quickly deploy a blank VM for development is great, too. It means you can give people more control over things. If you have lots of dev sites used for demos, there's the danger that someone might make a mistake editing httpd.conf and bring it all down, but if they're all in separate VMs, you don't need to worry.
The ability to (relatively) quickly deploy a blank VM for development is great, too. It means you can give people more control over things. If you have lots of dev sites used for demos, there's the danger that someone might make a mistake editing httpd.conf and bring it all down, but if they're all in separate VMs, you don't need to worry.
How does this work with Xen ? I haven't tried performance isolation yet. Is there a way to allocate CPU weights, guarantee minimum CPU cycle to VM's etc. Resource hogging by one VM could be really problematic in a hosting environment.
I think there is, although all I've been doing is limiting how many CPUs a VM is allowed to use. Every VM starts out with vcpus=1, and we add more when the load factor gets up too high.
This is quite effective on a server with 8 cores (two quad Xeons) and 4G of RAM, as by the time you've given each VM 512M of RAM, there's plenty of cores to go around. It would be more problematic on a single or dual core machine, though.
There are still a couple of ways a single VM can kill the box, though: it can max out the disks (so you really don't want to be running out of memory, unless you have a dedicated swap drive, so swap doesn't compete with normal disk access) and it can max out the network bandwidth.
In a hosting environment you'd probably want to set up some sort of internal traffic shaping so the each VM has a fair go at the available bandwidth, and you'd also want to keep swap partitions on a separate disk to data. This way if a couple of VMs run out of memory, they compete for the swap disk but other VMs that are within their limits and don't touch the swap disk don't see a problem.
I don't know, but I imagine you might be able to do some magic with LVM to balance disk transactions/throughput between logical volumes...
This is quite effective on a server with 8 cores (two quad Xeons) and 4G of RAM, as by the time you've given each VM 512M of RAM, there's plenty of cores to go around. It would be more problematic on a single or dual core machine, though.
There are still a couple of ways a single VM can kill the box, though: it can max out the disks (so you really don't want to be running out of memory, unless you have a dedicated swap drive, so swap doesn't compete with normal disk access) and it can max out the network bandwidth.
In a hosting environment you'd probably want to set up some sort of internal traffic shaping so the each VM has a fair go at the available bandwidth, and you'd also want to keep swap partitions on a separate disk to data. This way if a couple of VMs run out of memory, they compete for the swap disk but other VMs that are within their limits and don't touch the swap disk don't see a problem.
I don't know, but I imagine you might be able to do some magic with LVM to balance disk transactions/throughput between logical volumes...
Xen is great for many things, especially for performance. With help from newer processors it can essentially run a virtualized OS at close to native speeds. You could do the same with commercial offerings from VMWare but as far as free solutions go it doesn't get much better than this. Having separate VMs for each service does indeed make it easy to scale but you don't need to go through this overhead from the start. Instead try to focus on building share nothing architectures that can scale horizontally and then you can worry about VMs at a later point.
What you build should be scalable, whether you are on one server or a thousand. I'm a huge VM fan, mainly because of the complexity of some systems we use. If you solve your app for a few VMs somewhere, it becomes trivial (mostly) to scale. Then you can do backups easily, share snapshots of complex systems, allocate hardware resources virtually, etc.
Not sure about Xen, though. I've been using VMWare for a bit and I'm a happy customer.I would be interested in hearing how the various VM tools compare.
Not sure about Xen, though. I've been using VMWare for a bit and I'm a happy customer.I would be interested in hearing how the various VM tools compare.
We're running ESer.org (http://www.eser.org/usa) on over 120 different virtualized linux servers.
Moving a virtual machine from one server to another involves a simple rsync operation and takes just a few minutes over gigabit ethernet. Same goes for cloning virtual machines.
Moving a virtual machine from one server to another involves a simple rsync operation and takes just a few minutes over gigabit ethernet. Same goes for cloning virtual machines.
I just looked at eser.org. Neat-o. Have you thought about adding a comments/ratings system or features matrix?
There is a comments system on every page. Just scroll down to the bottom, type in your comment, and hit submit!
I should probably make that feature more obvious.
I should probably make that feature more obvious.
Yes, thanks. I see that now. Perhaps if you made the fonts smaller and separated the questions from the answers?
Definitely. Good suggestion. I didn't think of that. Thanks.
Unfortunately, I'm more of a low-level kernel hacker and math geek (machine learning algorithms, cluster file systems, robotics, etc..) than I am a UI person.
Which is a _major_ problem since the first thing people see is the UI.
And since I wrote my own web server and database, it wasn't really possible to use an off-the-shelf pretty-looking CMS.
So, my current project is a web-based satistical datamining system. And the UI is ... a text box. Which means it will be more difficult for me to f-up the UI. Although, I still might manage to do that.
Unfortunately, I'm more of a low-level kernel hacker and math geek (machine learning algorithms, cluster file systems, robotics, etc..) than I am a UI person.
Which is a _major_ problem since the first thing people see is the UI.
And since I wrote my own web server and database, it wasn't really possible to use an off-the-shelf pretty-looking CMS.
So, my current project is a web-based satistical datamining system. And the UI is ... a text box. Which means it will be more difficult for me to f-up the UI. Although, I still might manage to do that.
Actually, there's a good reason why the UI is so odd:
* My web-browser of choice is Lynx :-)
* My web-browser of choice is Lynx :-)
VM is generally considered for servers, but I'm interested for desktop clients.
I would like to have one VM for off network developments or document editing where I can ensure viruses, spyware or trojans are unable to contaminate the system.
I would like a VM for web surfing, where I could easily throw away the VM and restart with a fresh one. Same for mail or other internet applications.
I would also like to have one VM dedicated to do security sensitive tasks like monitoring my bank accounts and so.
Now browsing reddit can result in installing a trojan or whatever that puts all my applications in danger.
Pushing this logic a bit further, we can see that the logic is to go for a more lightweight desktop with eventually more smartness on the remote service than today. But also to combine multiple OS on a same machine. Such systems would work as well on lightweight hand held devices as on big PC with 128 CPUs.
I would like to have one VM for off network developments or document editing where I can ensure viruses, spyware or trojans are unable to contaminate the system.
I would like a VM for web surfing, where I could easily throw away the VM and restart with a fresh one. Same for mail or other internet applications.
I would also like to have one VM dedicated to do security sensitive tasks like monitoring my bank accounts and so.
Now browsing reddit can result in installing a trojan or whatever that puts all my applications in danger.
Pushing this logic a bit further, we can see that the logic is to go for a more lightweight desktop with eventually more smartness on the remote service than today. But also to combine multiple OS on a same machine. Such systems would work as well on lightweight hand held devices as on big PC with 128 CPUs.
I haven't actually gotten around to trying this out and getting it working, but Enomalism.com claims to have open source software that lets you to manage your VMs and ease deployment between VMWare and AWS EC2. Seems too good to be true.
I've been using VMWare on the desktop for over a year. It's just good software, and a joy to develop on.
I can develop on whatever OS I want, and not have to worry about multiple monitor support, or other sticky install issues. I don't have to worry about not having a decent Office suite.
I can set up a server on a VM, poke around on it, and throw it away and start over. I can clone a server and try different setups that vary from the base. I can move a server from one box to another without problems.
When I get to the point of having my apps out there, I'm certainly going to have my servers virtualized. It's a no brainer for me.
If you want to try it out, VMware server is a free dowload. http://www.vmware.com/products/server/
I can develop on whatever OS I want, and not have to worry about multiple monitor support, or other sticky install issues. I don't have to worry about not having a decent Office suite.
I can set up a server on a VM, poke around on it, and throw it away and start over. I can clone a server and try different setups that vary from the base. I can move a server from one box to another without problems.
When I get to the point of having my apps out there, I'm certainly going to have my servers virtualized. It's a no brainer for me.
If you want to try it out, VMware server is a free dowload. http://www.vmware.com/products/server/
I have started using Xen and like it so far. There are a few glitches, like Xen will hide memory from the dom0 and not give it back without a reboot, etc. Overall however I would recommend Xen, especially if you want Linux.
For really robust, high-performance stuff I heartily recommend Solaris and Zones. CPU impact is 1%, filesystem performance difference is practically zero. If you use packages from Blastwave you can have almost all of the software you care about installed.
For really robust, high-performance stuff I heartily recommend Solaris and Zones. CPU impact is 1%, filesystem performance difference is practically zero. If you use packages from Blastwave you can have almost all of the software you care about installed.
We've built our development infrastructure with virtualization technologies. Our main development server is a Qemu image which runs as a process on my desktop under linux kernel virtualization (KVM). Inside that server, various resources we use to organize development such as subversion, trac, irc, build system, etc. are isolated into separate vserver partitions. Later when we need to move everything onto dedicated server(s) it's clear and obvious how to do it.
If your services are already parallelizable to the degree that they could sit on separate VM's, couldn't you just run them side by side on the same machine without all of the extra overhead a VM gives you? Maybe this is a Rails-related problem, but in general the hard part is not the move from parallelized tasks on one machine to many. The big problem is taking some non-parallel code and trying to distribute it much later on.
In this case we're really talking about configuration isolation. If you're running a database and a Web server on one machine and then want to move them onto separate machines, you have to untangle and move the config files. But if they're separate VMs from the start, then there's nothing to untangle.
Again, the process of getting your services to the level that they can run independently (read: non-tangled config files) means that you don't need separate VM's.
Anyone know a good Xen configuration tool? There doesn't seem to be a good equivalent of say, VMware Server for Linux.
Don't care whether it's Open source or proprietary, just want something that makes the image creation, starting / stopping, networking and reconfig simple.
Don't care whether it's Open source or proprietary, just want something that makes the image creation, starting / stopping, networking and reconfig simple.
Virtualmin, Inc. has just released a product called VM2 (Virtualmin Machine Manager) into private beta. It manages Xen, Solaris Zones, Linux vzervers, and Amazon EC2 virtualized systems, as well as physical systems, from a central management interface. It's pretty nifty.
It'll be public in a couple of weeks, tops.
It'll be public in a couple of weeks, tops.
XenSource released a product called XenCenter that looked similar to the VMWare Server Console. Can watch a demo after signing up to receive their spam: http://www.xensource.com/Pages/XenEnterprise_Demo.aspx
Note that it's written on .NET and is Windows only.
Note that it's written on .NET and is Windows only.
crap, I have been using it at work for the last couple of months. The console doesn't refresh correctly and they prob. went the dot-net way to impress microsoft. Then they got bought by Citrix. Overall, I feel that the value-add on top of the open source xen is rather weak, to technically savvy users at least.
Anyone that is using EC2 is using a customized version of zen. We have started the process to move our services over to ec2.
first, vmware, and then we plan to make AMIs for ec2.
first, vmware, and then we plan to make AMIs for ec2.
I'm just curious whether anybody's tried hosting their app in a virtualized environment like this, as well as what people's thoughts are about this in general. Is it a good idea? Is Xen stable enough and is the performance good enough for this to work? Is it premature optimization for a startup not having scalability issues yet to be doing something like this, or do the other benefits help make it more justifiable? Would this really be all that much more work to set up compared to running all the services on a single box anyway?