I left Heroku for OpsWorks and shaved 40% off response times(stefanwrobel.com)
stefanwrobel.com
I left Heroku for OpsWorks and shaved 40% off response times
http://www.stefanwrobel.com/heroku-to-opsworks
9 comments
I'd be really interested to know more about how the switch shaved 40% off of the response times. What was the major bottleneck? RAM? CPU? Network IO to PostgreSQL? Network IO to memcache?
Heroku is offering 2X and soon, 4X dynos; did he experiment with these?
The author also sounds rather excited about nginx serving static assets. The Rack::Cache problem on Heroku is easily solved with a CDN layer (and Heroku even recommends you do so.) (Cult Cosmetics /is/ set up with CloudFront, so the author obviously knows this.)
There are a lot of Rails architecture smells in this article, such as worrying about the 60-second boot timeout and the 30-second request timeout. [1] I have to wonder if the time invested in learning Chef and managing the architecture may have been spent making the app more performant.
Developer time is really, really expensive. Not only can you bill $100-$150 hour instead of learning how to use Chef, you could be shipping features for your own products, which can bring you passive income forever. A lot of companies are successfully running on Heroku, and it seems strange to throw everything out for the explained reasons.
[1] Specifically, adding 'require: false' to your Gemfile and only loading heavy libraries when you need them greatly helps with boot time. Ideally, you won't run any image processing libraries (like RMagick) except in background jobs, and your main app should not even need these libraries available. Same goes for PDF generation, scraping technology, or any other heavy processing.
The 30-second timeouts either need to be handled via background jobs (the author mentioned they use sidekiq) or uploading directly to S3, side-stepping the web process entirely.
Heroku is offering 2X and soon, 4X dynos; did he experiment with these?
The author also sounds rather excited about nginx serving static assets. The Rack::Cache problem on Heroku is easily solved with a CDN layer (and Heroku even recommends you do so.) (Cult Cosmetics /is/ set up with CloudFront, so the author obviously knows this.)
There are a lot of Rails architecture smells in this article, such as worrying about the 60-second boot timeout and the 30-second request timeout. [1] I have to wonder if the time invested in learning Chef and managing the architecture may have been spent making the app more performant.
Developer time is really, really expensive. Not only can you bill $100-$150 hour instead of learning how to use Chef, you could be shipping features for your own products, which can bring you passive income forever. A lot of companies are successfully running on Heroku, and it seems strange to throw everything out for the explained reasons.
[1] Specifically, adding 'require: false' to your Gemfile and only loading heavy libraries when you need them greatly helps with boot time. Ideally, you won't run any image processing libraries (like RMagick) except in background jobs, and your main app should not even need these libraries available. Same goes for PDF generation, scraping technology, or any other heavy processing.
The 30-second timeouts either need to be handled via background jobs (the author mentioned they use sidekiq) or uploading directly to S3, side-stepping the web process entirely.
I was using 2X dynos on Heroku. I did have memory issues and had to continually reduce the number of unicorns that I was running. I think CPU and disk were the major bottlenecks, but again, Heroku makes it hard to track down exactly what the cause is.
Also, we're an ecommerce site so we rely on a lot of external services (specifically for payments) where we have to run the call in the web process ... hence the issue with 30s timeouts.
Also, we're an ecommerce site so we rely on a lot of external services (specifically for payments) where we have to run the call in the web process ... hence the issue with 30s timeouts.
> we have to run the call in the web process
I know you probably have good reason for this, but this smells like an architecture problem. Whenever you are talking to an external service it is a good idea to have your code laid out such that you can do that in the background. I understand that if you didn't plan enough architecture and maybe even product design to make that feasible from the get-go it is an expensive thing to build, but regardless of who is your hosting provider sooner or later you'll likely need to do it.
One thing I've always liked about heroku, at least in concept, is that it forces you to have good architecture. Some times it is a bit of a pain, but I think this pays off quicker than you seem to realize.
I know you probably have good reason for this, but this smells like an architecture problem. Whenever you are talking to an external service it is a good idea to have your code laid out such that you can do that in the background. I understand that if you didn't plan enough architecture and maybe even product design to make that feasible from the get-go it is an expensive thing to build, but regardless of who is your hosting provider sooner or later you'll likely need to do it.
One thing I've always liked about heroku, at least in concept, is that it forces you to have good architecture. Some times it is a bit of a pain, but I think this pays off quicker than you seem to realize.
How would you do payment processing in the background without temporarily storing the payment details?
I work for a payments company.
Generally, you use JavaScript to actually tokenize the card. You then save that token in your actual application, and then use it to actually charge against.
This model works just fine with background jobs.
Generally, you use JavaScript to actually tokenize the card. You then save that token in your actual application, and then use it to actually charge against.
This model works just fine with background jobs.
valid point.
I agree with the CDN comment. Please use a CDN to serve your assets no mater what you host on. It's so easy, and the performance gains to the end user are good, it should be an obvious choice.
Agree with both of you 100%, but the fact that Heroku forces this choice is a frustrating one nonetheless.
If companies think Heroku is more expensive they are failing to factor in how expensive it would be to have an employee to do all the work that Heroku takes off their plate. I work 45 - 50 hours a week developing. If I have to do devops - provisioning, deployment and system administration work - to maintain a server infrastructure there goes 15 - 20 hours! Surely my company is paying less to Heroku to manage that redundant, and at this point formulaic infrastructure than to me to do the same things that they've automated away. I can then work on our products. That seems less expensive, not more.
Once you have done the setup once and have it all automated, it's more like 1 hour per week keeping on top of upgrades, backups etc. So for many people heroku is an expensive option, even when starting out. Nothing wrong with using it, but outsourcing your infrastructure does not come cheap, compared to other options like a VPS.
VPS is much more expensive because you have to do all the work to provision (either manually or automatically) the software stack you are working with, configure every new instance either, configure your deployment and testing pipeline using a CI server and using git hooks, and keep the whole edifice up and running. The time overhead is very large, and it ends up taking over as the major cost driver.
Having done most of the steps you mention above for multile clients, I'd contend the time overhead is not very large - most things can be automated, and once that is done an extra instance is just a matter of spinning up a server and deploying - also you can scale a very long way on one or two instances when the hardware is beefy enough (e.g. this website), backup, deploy, testing can all be automated. There are definitely some situations where a service like Heroku or AWS makes a lot of sense, but it can get expensive very quickly; you're trading time to setup/manage for higher ongoing costs forever. VPS is cheaper but requires some setup. That's a trade-off each company has to evaluate individually given the talent and money they have.
[deleted]
[deleted]
Agree with most of what you say, but adding dynos increases concurrency but doesn't do anything to reduce response times. Sometimes you just need a faster box.
Or a bigger (more RAM) box, I posted this to the Ruby Rouges mailing list on tail latencies: https://gist.github.com/schneems/8849116
Concurrency is only half the battle with tail latencies. Of course dedicated CPU helps as well :)
Concurrency is only half the battle with tail latencies. Of course dedicated CPU helps as well :)
Good stuff schneems, in my case, too little, too late.
Understood, can't be everything to every one. BTW thanks for sticking around and answering questions/engaging :)
gladly :)
Based on his mention of SSD-backed instances (c3.large) being only $100/mo, I looked at EC2 pricing again to see if it had become more competitive. Turns out it only has 2x16 gigs of storage and 1 physical core.
Looking at the c3.8xlarge instances, they're specced pretty much exactly like the servers I just built for $9k/each, except they have half the ram (60 vs. 128) and half the storage (2x320 vs 4x320). And they cost the same to rent for 6 months as it does to buy one, including coloing.
So, if you need more than a fraction of a physical server, EC2 still loses pretty hard on the pricing. It seems as though it has become competitive with Softlayer, though, at least.
EDIT: Ah, but if you pay $11.6k upfront to reserve it for three years, it reduces the monthly bill from $1728 to ~$370, which is a much better deal than Softlayer offers. That's actually a pretty reasonable premium compared to racking them yourself, though you lose flexibility and the hardware is worse than what you'd get yourself. Well played, Amazon.
Looking at the c3.8xlarge instances, they're specced pretty much exactly like the servers I just built for $9k/each, except they have half the ram (60 vs. 128) and half the storage (2x320 vs 4x320). And they cost the same to rent for 6 months as it does to buy one, including coloing.
So, if you need more than a fraction of a physical server, EC2 still loses pretty hard on the pricing. It seems as though it has become competitive with Softlayer, though, at least.
EDIT: Ah, but if you pay $11.6k upfront to reserve it for three years, it reduces the monthly bill from $1728 to ~$370, which is a much better deal than Softlayer offers. That's actually a pretty reasonable premium compared to racking them yourself, though you lose flexibility and the hardware is worse than what you'd get yourself. Well played, Amazon.
>Your servers are also likely 2-5 times as fast in the CPU department too.
FYI, you've been hellbanned, Sssnake.
They use slightly better processors for the C3s than I use (2x8 core 2.8 vs. 2x8 core 2.6). I'd assume that if you get the largest instance, you probably have the whole machine to yourself and don't have issues with bad neighbors, so it's probably actually slightly faster than mine.
FYI, you've been hellbanned, Sssnake.
They use slightly better processors for the C3s than I use (2x8 core 2.8 vs. 2x8 core 2.6). I'd assume that if you get the largest instance, you probably have the whole machine to yourself and don't have issues with bad neighbors, so it's probably actually slightly faster than mine.
Possible he's been hellbanned, but his latest post is the only dead one.
Ah you're right. Sorry if it was a false alarm, Sssnake.
It was you that's right, he has indeed been hell-banned.
Agh he has. That's a shame, his comments seem to above average. I wonder how that happened.
Look at the last post he made before he was hell-banned.
Opsworks is nice, and it's a good packaged solution. A large number of smaller apps would benefit from a switch. It's also more fairly priced.
That being said, opsworks is still based on chef, which is still cumbersome. For most people Ansible is far easier to deal with if they don't fit into the opsworks box. The overhead of starting from scratch with ansible can be made up for with the faster overall development time.
That being said, opsworks is still based on chef, which is still cumbersome. For most people Ansible is far easier to deal with if they don't fit into the opsworks box. The overhead of starting from scratch with ansible can be made up for with the faster overall development time.
Big thumbs up for ansible, I have had great results with it it in a number of different production environments, I think the docs are great, and the devs in the irc channel on freenode are very helpful.
If you are about to embark on the journey of provisioning for the first time, or if you are looking for something a little more straightforward than chef/puppet (in my opinion at least...), I highly recommend checking out Ansible!
If you are about to embark on the journey of provisioning for the first time, or if you are looking for something a little more straightforward than chef/puppet (in my opinion at least...), I highly recommend checking out Ansible!
Another thumbs up. I'm currently in the process of migrating away from CFEngine to Ansible and it's been a breeze.
Ansible: I was in the "Don't go there" camp a while ago - they documentation was horrible, the whole website a mess and lots of things weren't discoverable at all (people using ansible told me that they used google site search to find stuff in the manual).
I returned a couple of days ago to look at the current state of things and I have to admit that I was impressed. Technically I really liked ansible for quite some while. Now it even comes with decent documentation. I'd back the suggestion to check it out.
I returned a couple of days ago to look at the current state of things and I have to admit that I was impressed. Technically I really liked ansible for quite some while. Now it even comes with decent documentation. I'd back the suggestion to check it out.
The doc is indeed pretty good, and available offline (ansible-doc). I quite like that it uses a very declarative syntax, and it's relatively easy to get idempotent operations. And the fact that you don't need to install anything on the machine you administer is definitely a strong point.
One downside is that I can see how tracking down role dependencies would be tricky if you had many different roles. I also haven't tried at scale.
One downside is that I can see how tracking down role dependencies would be tricky if you had many different roles. I also haven't tried at scale.
I was really excited by ansible, but I quickly found out that using `register: result` and `when: result|success` throughout so many playbooks meant I was back to the mess that is understanding the flow of a chef script that you didn't personally write.
Maybe that's not a problem since I rarely found an open source ansible playbook for anything I needed and had to always write my own.
Maybe that's not a problem since I rarely found an open source ansible playbook for anything I needed and had to always write my own.
Hi, I've published some reusable roles for ansible. They are still quite basic, but I'd love for you to have a look at them. https://github.com/search?q=%40zenoamaro+ansible
Always great to see people putting up open-source projects like this. I'll definitely give them a look, and at the least, get a grip on how other people found solutions with Ansible. Thanks man.
"That being said, opsworks is still based on chef, which is still cumbersome."
re:chef
I don't think this is necessarily a fair assessment and true for all. There are a ton of community cookbooks out there, and a very large community now online to help people with Chef. I've been using Chef for years and like anything else once you get over the learning period, it becomes second nature. Any tool that you use will have some learning curve to it, and so the trade offs that make it cumbersome for some, make it easy for others.
Also, you could technically Chef in Ansible in OpsWorks(if OpsWorks met other needs for you), but thats just crazy talk haha
re:chef
I don't think this is necessarily a fair assessment and true for all. There are a ton of community cookbooks out there, and a very large community now online to help people with Chef. I've been using Chef for years and like anything else once you get over the learning period, it becomes second nature. Any tool that you use will have some learning curve to it, and so the trade offs that make it cumbersome for some, make it easy for others.
Also, you could technically Chef in Ansible in OpsWorks(if OpsWorks met other needs for you), but thats just crazy talk haha
Seems like Opsworks does a lot more than just the provisioning, no?
"AWS OpsWorks can scale your application using automatic load-based or time-based scaling and maintain the health of your application by detecting failed instances and replacing them. You have full control of deployments and automation of each component."
How would you suggest using Ansible to cover that side of things?
"AWS OpsWorks can scale your application using automatic load-based or time-based scaling and maintain the health of your application by detecting failed instances and replacing them. You have full control of deployments and automation of each component."
How would you suggest using Ansible to cover that side of things?
Autoscaling groups with ansible-pull + cloud init to configure instances as they are spun up by AWS is what I've had the most success with. So your autoscaling group spins up a new instance using a prebaked AMI, and on first boot cloud-init will run ansible-pull locally to grab the latest configs and start up services as needed. Alternatively you can just use ansible to bake a new AMI as needed and skip the ansible-pull + cloud-init part altogether.
This just introduces additional tools/products to the stack, but you can use Ansible with the Netflix OSS stack, such that Ansible provisions your AMIs and Asgard manages auto-scaling and deployment (with very fine-grained control).
Good to know! I've been meaning to look into Ansible & SaltStack, and you gave me the kick in the butt I needed.
EDIT: ohai Andrew! Thanks for the comment.
EDIT: ohai Andrew! Thanks for the comment.
Nice writeup. I recently switched from Nodejitsu over to Heroku for my open source project gitignore.io and I noticed there was a slight slowdown in response time. The app score index in New Relic went from around 92 to 100 though so I guess Heroku's servers on AWS were a bit more reliable. I also added in a CDN to increase the speed of serving the static assets.
All in all, I can say that I'm happy with the transition, but I'm also running one Dyno which I'm far from maxing out. I feel like for a larger product, Heroku would be more problems down the line than it's worth.
All in all, I can say that I'm happy with the transition, but I'm also running one Dyno which I'm far from maxing out. I feel like for a larger product, Heroku would be more problems down the line than it's worth.
Exactly how I felt, great at small scale, but increasingly painful to use as you reach scale.
By the way, great tool, thanks for building it!
By the way, great tool, thanks for building it!
One limitation with OpsWorks is that as far as I know, there's no option to use spot instances whereas it's not a problem with EC2 launch configurations for AutoScaling Groups. Also, ASGs and OpsWorks both have options for time and load-based scaling with different interfaces.
It's easy enough to understand how this happened as OpsWorks came about via an acquisition, but consistency would certainly be nice.
Presenting overlapping yet incompatible features under the blanket of AWS is confusing and potentially pretty frustrating.
It's easy enough to understand how this happened as OpsWorks came about via an acquisition, but consistency would certainly be nice.
Presenting overlapping yet incompatible features under the blanket of AWS is confusing and potentially pretty frustrating.
FYI OpsWorks runs their Chef on Ruby 1.8, make sure your cookbooks passes on 1.8, and it doesn't depend on +1.9 :)
This is nothing on AWS OpsWorks, as they greatly simplified interface into managing clusters (for deployments, organization).
I found about this recently as I updated a community cookbook(python), as the Chef community is moving forward (with Chef 11+) that 1.8 has reach EOL and 1.9 is stable (2015) and 2.0 current, and 2.1 is new-hot-ness.
This is nothing on AWS OpsWorks, as they greatly simplified interface into managing clusters (for deployments, organization).
I found about this recently as I updated a community cookbook(python), as the Chef community is moving forward (with Chef 11+) that 1.8 has reach EOL and 1.9 is stable (2015) and 2.0 current, and 2.1 is new-hot-ness.
What do Opswork, Chef, and the Heroku platform have to do with response time?
Did the author investigate why the response time were that slow to begin with?
Did the author investigate why the response time were that slow to begin with?
I wouldn't say they were "slow to begin with" but improved speed is always nice. I think the major contributing factor was access to newer, more powerful instances. Heroku Dynos run on very old EC2 instances.
At WeHostels we moved from Heroku to Rackspace last year and we shaved ~30% off response times.
Curious from a user experience perspective would you consider going back to try out the PX dyno service? Do you use chef or puppet or any other software to manage your Rackspace instances?
A friend of mine and I have been working on a startup for the last 8 months. It has been tough as we have been working full time for a company and hacking like crazy once we got home, or on weekends.
We chose to use Heroku because of the dead simple workflow that would allow us to spend time developing features instead of maintaining the infrastructure. This may sound like a cliché, but boy was it a good decision. The platform is rock solid, deployments are a bless and the add-on system is just one of the best time savers you can get.
We know OpsWorks (or earlier Scalarium) from our full-time job, and we know how cool it is. But we also know that it requires a lot of time and additional effort to keep things running.
This may pay off if you run a company which can afford it, but you should think about it twice if you have limited resources (and other, more important things to do!).
Even now, after we have launched, we still see no reason why we should move from Heroku to OpsWorks (or another platform). We have made some optimizations and clever (to us) decisions to avoid running into scalability problems. Maybe we will think about migrating once we get a ton more clients and even more traffic. It is cheaper to just add some dynos than spending time on server administration.
In the end, it's all about making the right decision, so don't hurry and if you think Heroku is too expensive for your business, maybe you are not making enough revenue from your customers. On the other hand, if you have the money and time, than go for it.