Ask HN: What Linux distro for a Rails app that needs to scale?
26 comments
Linux distribution is probably the least important item in terms of scalability.
If you're talking about switching to nginx from Apache, that makes me think that you are still proxying to a collection of mongrels? I'd recommend switching to Phusion (aka mod_rails) as it seems to handle high loads much better.
If you're talking about switching to nginx from Apache, that makes me think that you are still proxying to a collection of mongrels? I'd recommend switching to Phusion (aka mod_rails) as it seems to handle high loads much better.
Debian. It's what Ubuntu is based on, but it's less crufty. I use Debian for everything I do, and I've never had a problem. Take my advice with a grain of salt however, as I've never really had to scale anything.
One other thing, make sure you build everything from the source. Get exactly what you want/need, and nothing more as well as the latest packages.
One other thing, make sure you build everything from the source. Get exactly what you want/need, and nothing more as well as the latest packages.
One other thing, make sure you build everything from the source.
I was right there with you until this bit of crazy.
Do not build everything from source. You're just asking for trouble when you do. You lose so many really important features of modern Linux distros when you do this, that I can't believe people are still suggesting it as a sane practice for production servers.
If you install from source you won't be able to update quickly and easily via apt-get or yum when a security update comes out...you'll have to rebuild. But, you probably won't even know a security update exists until it's too late. If you don't keep copious and specific notes about what your custom build does, and why you did it that way, you will forget a flag or overwrite or lose a config file, and it will break something next time you roll out a new version.
The performance benefits are probably negligible, and in my many years experience cleaning up amateur maintained systems, it is just as likely to be negative as it is positive. You probably know dramatically less about the package you're building than the maintainer of the OS package, and you're more likely to make mistakes than they are. Those mistakes could have security implications, performance implications, or stability implications.
Certainly, if you are the maintainer of some piece of software, you might want to replace it. I've never run a system without at least a few custom builds...but the custom pieces are usually software that I am one of the core maintainers of, and I know it better than just about anybody. And, when I do run custom software, I tend to build an RPM or deb of it, and setup a local yum or apt-get repo, so that when I need to replicate my production machine, I can do it quickly and consistently.
I was right there with you until this bit of crazy.
Do not build everything from source. You're just asking for trouble when you do. You lose so many really important features of modern Linux distros when you do this, that I can't believe people are still suggesting it as a sane practice for production servers.
If you install from source you won't be able to update quickly and easily via apt-get or yum when a security update comes out...you'll have to rebuild. But, you probably won't even know a security update exists until it's too late. If you don't keep copious and specific notes about what your custom build does, and why you did it that way, you will forget a flag or overwrite or lose a config file, and it will break something next time you roll out a new version.
The performance benefits are probably negligible, and in my many years experience cleaning up amateur maintained systems, it is just as likely to be negative as it is positive. You probably know dramatically less about the package you're building than the maintainer of the OS package, and you're more likely to make mistakes than they are. Those mistakes could have security implications, performance implications, or stability implications.
Certainly, if you are the maintainer of some piece of software, you might want to replace it. I've never run a system without at least a few custom builds...but the custom pieces are usually software that I am one of the core maintainers of, and I know it better than just about anybody. And, when I do run custom software, I tend to build an RPM or deb of it, and setup a local yum or apt-get repo, so that when I need to replicate my production machine, I can do it quickly and consistently.
What do you mean less crufty? I think this is just your perceived vision of Debian.
We use Ubuntu 8.04 LTS because it is committed to be supported for a very long time. Traditionally Debian releases have been supported with security updates and the like for one year after the subsequent release. There's no guarantee or date for Debian. Ubuntu 8.04 Server Edition on the other hand is supported through 2013... and that very well might turn out to be longer than Debian 5.0.
We use Ubuntu 8.04 LTS because it is committed to be supported for a very long time. Traditionally Debian releases have been supported with security updates and the like for one year after the subsequent release. There's no guarantee or date for Debian. Ubuntu 8.04 Server Edition on the other hand is supported through 2013... and that very well might turn out to be longer than Debian 5.0.
I would say building from source is a horrible idea if you want to scale. For us we had some sweet custom built from source setups that were fast as hell, but once we started putting more servers online it was a hassle to set each one up identically. Better to go with a standard install and be able to reproduce it easily - because no matter what you efficiency gains by compiling are, if you need a new server you will need a new server.
And if the answer is "visualized it" - then compiling is a waste of your time to being with. Better to run native from packages than virtualized from source.
And if the answer is "visualized it" - then compiling is a waste of your time to being with. Better to run native from packages than virtualized from source.
Time is probably your most important resource here. Shaving a few nanoseconds off a query because you compiled your kernel with one different option won't do you much good, whereas an easy to administer OS with readily available binaries will save you a lot of time and headache.
I'd say 64-bit Debian. I have already mentioned my qualms with Ubuntu elsewhere on HN (I don't like upgrading every 6 months, and I don't want to be stuck with old software in case I decide to skip an upgrade).
I'd say 64-bit Debian. I have already mentioned my qualms with Ubuntu elsewhere on HN (I don't like upgrading every 6 months, and I don't want to be stuck with old software in case I decide to skip an upgrade).
The Linux distribution isn't the big bottleneck, but yes, I'd probably go with the 64-bit Ubuntu Server.
I think Engine Yard, which is where I'm hosting right now, uses Gentoo, it seems to work fine.
I think Engine Yard, which is where I'm hosting right now, uses Gentoo, it seems to work fine.
Note about the distribution not being the big bottleneck: This is exactly right.
On the other hand, if you use a distribution that somehow 'benchmarks' marginally better but you don't know how to configure it or deal with it properly, you'll end up losing any performance gains you otherwise would have had. Better to deal with a distribution that works the way you do (Which is why a lot of people like CentOS or Ubuntu Server) than end up making a major mistake somewhere because you are in unfamiliar territory.
On the other hand, if you use a distribution that somehow 'benchmarks' marginally better but you don't know how to configure it or deal with it properly, you'll end up losing any performance gains you otherwise would have had. Better to deal with a distribution that works the way you do (Which is why a lot of people like CentOS or Ubuntu Server) than end up making a major mistake somewhere because you are in unfamiliar territory.
If you're scaling horizontally, is 64 bit a win? It eats up memory in Ruby processes. We switched our app (which does not need to scale; it's shrinkwrap) to 32 bit Ubuntu and that was a win for us.
(Gentoo? Blech. -funroll-loops.)
(Gentoo? Blech. -funroll-loops.)
We've noticed that several VPS providers are only offering 64-bit OS installs...which makes no sense. I kinda suspect it is simply a way to force people to pay for more memory down the road. The memory usage of dynamic languages (Ruby included) is dramatically higher in 64 bit builds...more than 50% in many cases.
If you don't have more than 4GB of RAM, it makes absolutely no sense to have a 64 bit OS install.
If you don't have more than 4GB of RAM, it makes absolutely no sense to have a 64 bit OS install.
Depends on the app I guess, we do a lot of large number computations that surpass the 32-bit limit, so I think for our specific case a 64-bit distro is much better.
More than the actual distro I'd say look into what kernel you are loading on there. Make sure you're running a nice tight kernel with any extra fluff turned off (or loading as modules).
I can't find the papers off hand but there has been some research into how kernel level cache tweaks will dramatically effect how web servers and DB servers will perform.
I can't find the papers off hand but there has been some research into how kernel level cache tweaks will dramatically effect how web servers and DB servers will perform.
Gentoo is based more for the tinkerer and full-time sysadmin. If you're prepared to spend time nursing the system, compiling everything, tweaking packages/the OS itself for every bit of performance possible, the Gentoo is a good solution. Until you're talking about really big scaling though, it's probably more time effective to use another distro.
As far as the rest goes, it would be better to use Debian before Ubuntu, but there's a lot of other distros as well. RHEL powers many of the webs servers as does SuSE and Debian and Gentoo. Once you get past that, it's mostly personal preference.
Also, big plugs for nginx. It's definitely a very impressive software package.
As far as the rest goes, it would be better to use Debian before Ubuntu, but there's a lot of other distros as well. RHEL powers many of the webs servers as does SuSE and Debian and Gentoo. Once you get past that, it's mostly personal preference.
Also, big plugs for nginx. It's definitely a very impressive software package.
I have no stats in front of me, but I have always been made to understand that BSD's (namely FreeBSD) handle high traffic better than linux.
I personally use Debian for everything. I can't imagine the distro making a huge difference really. I've used Debian as a desktop os for some time, so when I needed a server os, it was easy to get going quickly.
Switching from Apache to Nginx made more of a difference in my case than anything else I could have done I believe.
I personally use Debian for everything. I can't imagine the distro making a huge difference really. I've used Debian as a desktop os for some time, so when I needed a server os, it was easy to get going quickly.
Switching from Apache to Nginx made more of a difference in my case than anything else I could have done I believe.
I don't know FreeBSD vs. Linux (and god do I ever not want to see that one hashed out), but let me assure you based on significant commercial experience in performance-critical apps that the other BSDs are not the way to go here.
You're working against the grain if you go BSD instead of Linux with Rails, whether you like FreeBSD or not.
You're working against the grain if you go BSD instead of Linux with Rails, whether you like FreeBSD or not.
but I have always been made to understand that BSD's (namely FreeBSD) handle high traffic better than linux
For the past several years, and up until FreeBSD 7 you've always been seriously misled. FreeBSD has historically had dramatically worse performance in most concurrent server applications (certainly in multi-process SMP workloads). As of FreeBSD 7, it generally is the equal or slight better of Linux. It is not better in enough categories or by a large enough margin for anyone to say with confidence that a particular workload will perform better on FreeBSD or Linux. One would always have to benchmark the actual application, and the differences would be minimal in most cases...and with every new kernel release, you'd have to benchmark again, because things change and performance advantages might flip from one release to the next.
And, tptacek is correct, none of the other BSDs has inherited these dramatic performance improvements found in FreeBSD 7, so they are very unlikely to provide better performance than any modern Linux version on modern multi-core hardware.
For the past several years, and up until FreeBSD 7 you've always been seriously misled. FreeBSD has historically had dramatically worse performance in most concurrent server applications (certainly in multi-process SMP workloads). As of FreeBSD 7, it generally is the equal or slight better of Linux. It is not better in enough categories or by a large enough margin for anyone to say with confidence that a particular workload will perform better on FreeBSD or Linux. One would always have to benchmark the actual application, and the differences would be minimal in most cases...and with every new kernel release, you'd have to benchmark again, because things change and performance advantages might flip from one release to the next.
And, tptacek is correct, none of the other BSDs has inherited these dramatic performance improvements found in FreeBSD 7, so they are very unlikely to provide better performance than any modern Linux version on modern multi-core hardware.
It's also worth noting that at least around christmas of last year, it was brought to light that the Ruby 1.8 package in apt-get on ubuntu is TWICE as slow as building 1.8 from source.
make sure you build from source.
http://antoniocangiano.com/2008/12/09/the-great-ruby-shootou...
make sure you build from source.
http://antoniocangiano.com/2008/12/09/the-great-ruby-shootou...
Gentoo if you have a lot of time to customize everything and don't mind rehacking on the kernel configuration with every release. Ubuntu if you want something that just works and has available commercial support if you need it down the road. Debian if you like Ubuntu but somehow feel that Debian is more "pure". I wouldn't use anything else.
Gentoo is cargo-cult Linux. Tweaking compiler options and use flags rarely produces any measurable benefit, but it guarantees that you're using a unique, untested configuration that may expose bugs other people have never seen.
Cannot upvote this enough. Gentoo is a fools game on servers. If you want to dick around several hours every time you update your system on your desktop machine, have at it. But, if you're inviting that kind of chaos into your life on your production server that provides your livelihood, you deserve no pity (and you're unlikely to get any from anyone who has enough experience to be managing a server).
I have a vested interest in being pretty much OS agnostic; our products support tons of systems and versions, and you can easily get me to say nice things about all sorts of operating systems. But, I simply can't have no opinion about Gentoo. It's just such a poor choice for a server that I'd feel guilty not using the strongest possible language to discourage folks from using it.
I have a vested interest in being pretty much OS agnostic; our products support tons of systems and versions, and you can easily get me to say nice things about all sorts of operating systems. But, I simply can't have no opinion about Gentoo. It's just such a poor choice for a server that I'd feel guilty not using the strongest possible language to discourage folks from using it.
Been through the horror over here, on Ubuntu now. No measurable difference in speed of the server, but plenty measurable difference in speed of deploying new ones and a major decrease in swearing from the direction of the server room.
Whatever you're comfortable with. The distro really won't be the thing that limits scaling. In fact, I'd be surprised if there was a consistently measurable difference between them.
good q. if you can, use the same thing on the server as you use on your desktop. i use 32 bit ubuntu (8.04). if it performs fine on your 2GB RAM thinkpad, it will probably do fine on an ec2 box, no? if the config works on my desktop, it will work on the server, etc. tremendous advantage in terms of sysadmin requirements in my book. your mileage may vary depending on how much you like os x :>)
Definitely Red Hat for anything serious that's worth the support costs. Else you can settle for CentOS, which is Red Hat minus the support.
Use whatever you know your way around best. If you're comfortable with openbsd and it can do everything you need, then go with that.
what's wrong with openbsd? is your webapp currently running on it?
i'm curious because i use openbsd+lighttpd (soon will be nginx) in my colo (because no slice/vps offers openbsd afaik)
i'm curious because i use openbsd+lighttpd (soon will be nginx) in my colo (because no slice/vps offers openbsd afaik)
While I love OpenBSD, I'm more concerned with scalability than security, at this point.
As of right now, I'm going to use nginx over Apache, and probably use a 64-bit Ubuntu server.
What would you use?