What to do when your Amazon EBS fails(lucidchart.com)
lucidchart.com
What to do when your Amazon EBS fails
https://www.lucidchart.com/techblog/2013/03/08/lesson-learned-amazon-went-down/
6 comments
I know several systems administrators who have spent a lot of sleepless nights dealing with SoftLayer networking outages over the last few months.
Sure, the original AWS sales pitch was a bit disingenuous, but it should be obvious to anyone with deployment experience that everything breaks all the time.
It's not about the "quality" (by what metric?) brand IMO - it's about about evaluating a provider, understanding the failure modes, and choosing whether you're going to try to engineer around them, accept some downtime/customer-facing failure, or find a provider with failure modes you like better.
This article is about picking AWS and understanding the EBS failure case, just like how if I picked SoftLayer I'd be trying to understand their network topology and power setup to see where failures might coordinate and try to understand what to do about it.
Sure, the original AWS sales pitch was a bit disingenuous, but it should be obvious to anyone with deployment experience that everything breaks all the time.
It's not about the "quality" (by what metric?) brand IMO - it's about about evaluating a provider, understanding the failure modes, and choosing whether you're going to try to engineer around them, accept some downtime/customer-facing failure, or find a provider with failure modes you like better.
This article is about picking AWS and understanding the EBS failure case, just like how if I picked SoftLayer I'd be trying to understand their network topology and power setup to see where failures might coordinate and try to understand what to do about it.
>> why do people chose to waste their very precious time writing software to deal with EBS failures (and other AWS deficiencies) instead of just not using AWS?
1) Proven Scale. Netflix. They're bigger than I will be until such time my company can afford to buy it's own data centers. At that point, I can re-evaluate. 2) Operational overhead. Startup, running hundreds of hosts, thousands of requests per second, 5 developers, 0 sys-engineers. 3) Lock in. We use Cloudwatch, Dynamo, RDS, Beanstalk, S3, EMR and more. We also use Heroku. No one else out there has the full suite of offerings. Not using AWS would require us to stitch together offerings from several vendors, some technology wouldn't be feasible having to hop across the internet (do you want the lag to get to your database from one data center to another?) 4) Billing. We get 1 Bill which we can predict. AWS takes credit cards, will do Net 30 billing, and are generally pretty flexible. 5) Customer Service. Don't get me wrong, support at AWS is ... not it's strong suit. We also don't spend enough to be a "big fish". However, if we used 10 different providers to get our infrastructure together, we'd me a minnow to each one. By consolidating, we're a small-medium sized customer which is enough to get into beta's, info on their roadmap ahead of time, and some personal service. 6) Excuses for downtime. So many high profile companies are on AWS that if we have an outage directly attributable to AWS, the odds are a lot of the rest of the internet is down, too. Seriously. It sounds stupid, but it's a huge reason. 2 years ago we were raising money. Right in the middle of our final week (when all our investors were visiting our site and making sure they really wanted to sign on the dotted line), we had an outage that lasted several days. This was the AWS outage that took out Heroku for a few days, and ours was one of the last databases to get restored. If we were using almost anyone else, investors might have been like "oh, this team doesn't really know what they're doing, they're not really that technically competent, I'm not going to invest". But since Netflix, Foursquare, and dozens of other companies were all offline at the same time, the investors were just like "they're using AWS, if Netflix can't keep online with hundreds of engineers, then I'm not going to expect that this little startup is going to do better". 7) Could I really do it better? Is anyone else doing better? AWS is the leader, so we all see the warts ... but at least they keep their certs updated (ahem microsoft)
1) Proven Scale. Netflix. They're bigger than I will be until such time my company can afford to buy it's own data centers. At that point, I can re-evaluate. 2) Operational overhead. Startup, running hundreds of hosts, thousands of requests per second, 5 developers, 0 sys-engineers. 3) Lock in. We use Cloudwatch, Dynamo, RDS, Beanstalk, S3, EMR and more. We also use Heroku. No one else out there has the full suite of offerings. Not using AWS would require us to stitch together offerings from several vendors, some technology wouldn't be feasible having to hop across the internet (do you want the lag to get to your database from one data center to another?) 4) Billing. We get 1 Bill which we can predict. AWS takes credit cards, will do Net 30 billing, and are generally pretty flexible. 5) Customer Service. Don't get me wrong, support at AWS is ... not it's strong suit. We also don't spend enough to be a "big fish". However, if we used 10 different providers to get our infrastructure together, we'd me a minnow to each one. By consolidating, we're a small-medium sized customer which is enough to get into beta's, info on their roadmap ahead of time, and some personal service. 6) Excuses for downtime. So many high profile companies are on AWS that if we have an outage directly attributable to AWS, the odds are a lot of the rest of the internet is down, too. Seriously. It sounds stupid, but it's a huge reason. 2 years ago we were raising money. Right in the middle of our final week (when all our investors were visiting our site and making sure they really wanted to sign on the dotted line), we had an outage that lasted several days. This was the AWS outage that took out Heroku for a few days, and ours was one of the last databases to get restored. If we were using almost anyone else, investors might have been like "oh, this team doesn't really know what they're doing, they're not really that technically competent, I'm not going to invest". But since Netflix, Foursquare, and dozens of other companies were all offline at the same time, the investors were just like "they're using AWS, if Netflix can't keep online with hundreds of engineers, then I'm not going to expect that this little startup is going to do better". 7) Could I really do it better? Is anyone else doing better? AWS is the leader, so we all see the warts ... but at least they keep their certs updated (ahem microsoft)
Some more items:
o Liquidity of engineering talent - thousands of engineers know how to immediately work with the AWS infrastructure
o Design Patterns - They are constantly being updated, discussed, improved upon. People share how to work around AWS issues.
o Commitment to reducing prices - AWS is constantly dropping the prices for everything.
o Commitment to increasing the number of services - AWS is constantly releasing new and useful services on their infrastructure.
o Likelihood of being in business in 3 years. AWS isn't going anywhere. You can rely on them being around.
o Really good post mortems. Each time they fail, they do pretty transparently, and deliver a really painful post mortem that encourages me to believe they are committed to constantly improving.
o Multiple Availability and Region zones. I can, with very little effort, make my services available in redundant regions. I don't think there has ever been a multi-region failure with AWS, and, at most, it's only happened a couple times in their history. Multi-Availability zone failures have happened (and Amazon has apologized - that shouldn't happen) - but, it's < half a dozen times.
o Market Leader Advantage - The market leader tends to win. Microsoft for Desktops, Oracle for Databases, Linux for simple Web/Database/PHP stacks. This isn't always the case (Juniper can beat Cisco in a lot of places) but you know Amazon will be a Top-3 contender for at least the next 5 years.
I realize a lot of these are "Business" reasons, which is incredibly frustrating for people who are comfortable with hosting bare metal, and realize that, given just half the money, they could go into a CoLo, rent a cabinet, buy some servers, routers, firewalls, storage - install it, and return twice the performance that they would get on AWS.
But quite often, the business argument is a more important driver for CxO people making decisions (and assigning budget) than technical reasons.
And, with all that said - I've been in about a dozen Tier-1 data centers in the last 15 years, and, with one exception, every one of them has had multiple sustained outages. (The exception has been Switch SuperNap in Las Vegas - 3 years and no major outage.) - So, failures are part of the business. I'm not expecting Amazon to eliminate them, so it doesn't disappoint me when they happen.
o Liquidity of engineering talent - thousands of engineers know how to immediately work with the AWS infrastructure
o Design Patterns - They are constantly being updated, discussed, improved upon. People share how to work around AWS issues.
o Commitment to reducing prices - AWS is constantly dropping the prices for everything.
o Commitment to increasing the number of services - AWS is constantly releasing new and useful services on their infrastructure.
o Likelihood of being in business in 3 years. AWS isn't going anywhere. You can rely on them being around.
o Really good post mortems. Each time they fail, they do pretty transparently, and deliver a really painful post mortem that encourages me to believe they are committed to constantly improving.
o Multiple Availability and Region zones. I can, with very little effort, make my services available in redundant regions. I don't think there has ever been a multi-region failure with AWS, and, at most, it's only happened a couple times in their history. Multi-Availability zone failures have happened (and Amazon has apologized - that shouldn't happen) - but, it's < half a dozen times.
o Market Leader Advantage - The market leader tends to win. Microsoft for Desktops, Oracle for Databases, Linux for simple Web/Database/PHP stacks. This isn't always the case (Juniper can beat Cisco in a lot of places) but you know Amazon will be a Top-3 contender for at least the next 5 years.
I realize a lot of these are "Business" reasons, which is incredibly frustrating for people who are comfortable with hosting bare metal, and realize that, given just half the money, they could go into a CoLo, rent a cabinet, buy some servers, routers, firewalls, storage - install it, and return twice the performance that they would get on AWS.
But quite often, the business argument is a more important driver for CxO people making decisions (and assigning budget) than technical reasons.
And, with all that said - I've been in about a dozen Tier-1 data centers in the last 15 years, and, with one exception, every one of them has had multiple sustained outages. (The exception has been Switch SuperNap in Las Vegas - 3 years and no major outage.) - So, failures are part of the business. I'm not expecting Amazon to eliminate them, so it doesn't disappoint me when they happen.
Serious question: You have hundreds of hosts, but you don't think the savings of colocating your own hardware won't pay for an engineer to take care of the machines?
It would probably be possible to do it cheaper, but there is tremendous cost advantage at scale in this business. Then there's the question of focus. Do I want to have to hire the ops folks? their managers? create new payscales and review standards? Am I going to be good at attracting or judging talent in this field that is only tangentially related to software that I do know? Will the best people in that field want to work for me?
Serious question -- how are alternatives so much better?
You've always got to "waste your very precious time" on something -- putting together servers, dealing with hard drive crashes, etc. No matter what, you have to plan for things breaking, and a lot of the time, they'll break in ways you couldn't plan for.
It's not like there are other cloud platforms out there with zero failures and zero deficiencies. But how, for example, is SoftLayer better than AWS?
You've always got to "waste your very precious time" on something -- putting together servers, dealing with hard drive crashes, etc. No matter what, you have to plan for things breaking, and a lot of the time, they'll break in ways you couldn't plan for.
It's not like there are other cloud platforms out there with zero failures and zero deficiencies. But how, for example, is SoftLayer better than AWS?
> But how, for example, is SoftLayer better than AWS?
Where do I begin... The latency to our dedicated DB servers is always 1-2 milliseconds. Never 5. Never 50. I do not understand how people tolerate 50ms response time from memcached servers on AWS.
We pay much less "per request". Way less. We never have to worry about servers disappearing. We don't need proprietary APIs for server images because the Chef script makes me a DB or a DNS box out of any Linux box.
We do not waste any time "architecting for failure": our databases run on RAID arrays with hot standbys, no need for chaos monkeys. Everything is very, very simple. We never used SoftLayer documentation because nothing of what we use needs to be documented.
And most importantly, we had less than an hour of downtime in 3 years. And none of that was Softlayer's fault.
It becomes much harder to reason about performance, uptime and responsiveness of your system when the underlying infrastructure does not provide guarantees on basic building blocks like I/O latency. Everything on AWS is always in flux and you have to over-allocate resources and/or over-engineer around those faulty building blocks. It may make you feel like a hero to survive in this jungle, but doesn't it look like a "forced premature optimization"?
99% of every graduating YC class could be running on a single pair of dedicated low-end servers and saving themselves a TON of time.
Where do I begin... The latency to our dedicated DB servers is always 1-2 milliseconds. Never 5. Never 50. I do not understand how people tolerate 50ms response time from memcached servers on AWS.
We pay much less "per request". Way less. We never have to worry about servers disappearing. We don't need proprietary APIs for server images because the Chef script makes me a DB or a DNS box out of any Linux box.
We do not waste any time "architecting for failure": our databases run on RAID arrays with hot standbys, no need for chaos monkeys. Everything is very, very simple. We never used SoftLayer documentation because nothing of what we use needs to be documented.
And most importantly, we had less than an hour of downtime in 3 years. And none of that was Softlayer's fault.
It becomes much harder to reason about performance, uptime and responsiveness of your system when the underlying infrastructure does not provide guarantees on basic building blocks like I/O latency. Everything on AWS is always in flux and you have to over-allocate resources and/or over-engineer around those faulty building blocks. It may make you feel like a hero to survive in this jungle, but doesn't it look like a "forced premature optimization"?
99% of every graduating YC class could be running on a single pair of dedicated low-end servers and saving themselves a TON of time.
> We do not waste any time "architecting for failure": our databases run on RAID arrays with hot standbys, no need for chaos monkeys.
Ha - your in for some fun the first time your hardware fails.
Ha - your in for some fun the first time your hardware fails.
We're not. Hardware failures were "invented" before Amazon marketing thought of them. There are reasonable engineering overhead to deal with them, and we do. You are welcome to go and unplug any of our severs with no measurable downtime or data loss.
My point is that we didn't have to spend nearly as much money and time as AWS forces you to do.
Another point I want to make: not all hardware is the same. If you're running in Amazon/Google style data center optimized around energy efficiency and real estate density and you're spending serious R&D money on tolerating server failures.
But guess what: you are not Google and you are not Amazon. You need to be in a datacenter which is optimized for individual server longevity. Cooler temperatures. Better hardware. Have a hot stand-by. That's cheap. Such way to "compute" may be expensive for Google scale to run, but it's much cheaper and easier for nearly everyone else.
My point is that we didn't have to spend nearly as much money and time as AWS forces you to do.
Another point I want to make: not all hardware is the same. If you're running in Amazon/Google style data center optimized around energy efficiency and real estate density and you're spending serious R&D money on tolerating server failures.
But guess what: you are not Google and you are not Amazon. You need to be in a datacenter which is optimized for individual server longevity. Cooler temperatures. Better hardware. Have a hot stand-by. That's cheap. Such way to "compute" may be expensive for Google scale to run, but it's much cheaper and easier for nearly everyone else.
Given your response - I don't understand why you would then make the statement about Chaos Monkey. It also appears that your earlier statement appears to be false: "We do not waste any time "architecting for failure"". If you can pull the plug on a server and not lose service, you certainly have architected for failure.
Chaos Monkey does effectively exactly what you claim to be able to survive - it unplugs servers. The fact that it runs on AWS is inmaterial, it exists to make sure you really can survive a server disappearing. The underlying point is that many teams build their software to survive server failures and never test the fail-over till it occurs for real, that often goes very badly. Chaos Monkey tests fail-over on a regular basis so you find out about it before it costs you hours of downtime.
Chaos Monkey does effectively exactly what you claim to be able to survive - it unplugs servers. The fact that it runs on AWS is inmaterial, it exists to make sure you really can survive a server disappearing. The underlying point is that many teams build their software to survive server failures and never test the fail-over till it occurs for real, that often goes very badly. Chaos Monkey tests fail-over on a regular basis so you find out about it before it costs you hours of downtime.
I always point to this article: http://www.jamiebegin.com/why-an-ec2-instance-isnt-a-server/
> People, AWS original sales pitch was to take care of scaling for you, not the other way around.
As far as I know, it was never about that, but about giving you the tools to spin-up whatever you need at will. The cool thing about AWS has always been the APIs.
I would rather take a service that has downtime but give me the tools to failover/recover than a service that promises something they cannot promise (100% uptime) and when it inevitably fails, I'm at the mercy of their support.
As far as I know, it was never about that, but about giving you the tools to spin-up whatever you need at will. The cool thing about AWS has always been the APIs.
I would rather take a service that has downtime but give me the tools to failover/recover than a service that promises something they cannot promise (100% uptime) and when it inevitably fails, I'm at the mercy of their support.
Went from an AWS emergency every month or so to nothing but smiles with Linode. AWS is the cloud ghetto. Yes, Linode isn't a platform like AWS, but I was just using EC2 and all the platform tricks like the RBS and whatnot really are a lazy man's "I'm too busy to install mysql on a box" excuse.
People who need instant scaling have limited alternatives, but not sure why everyone is sticking with AWS.
People who need instant scaling have limited alternatives, but not sure why everyone is sticking with AWS.
Point me to someone else who does latency based routing (roughly equivalent to GeoDNS) with data centres around the world and I'm there.
>The world is full of wonderful alternatives
Because those alternatives as just as bad.
>We've been running a very high traffic system there with individual box uptimes of 700 days before we had to reboot them during scheduled OS upgrade/maintenance.
So, just like EC2?
Because those alternatives as just as bad.
>We've been running a very high traffic system there with individual box uptimes of 700 days before we had to reboot them during scheduled OS upgrade/maintenance.
So, just like EC2?
Definitely not like Amazon US-East-1.
Which providers offer something equivalent to EBS, are worldwide, and have better reliability?
Worldwide is not everyone's requirement. Is Amazon reliable across the world? Sure. So is any other provider if they have more than one datacenter, or you use more than one provider and each provider is in different locations.
When the question is "why do people use X?", and people give totally reasonable answers, it doesn't make much sense to respond "but if we ignore your requirements then you could use someone else".
This is a total side question but I noticed this line:
> The process was stuck in the kernel waiting for an IOCTL, so killing the process did nothing – even kill -9. It was going to stay that way until EBS was back to normal.
I did not know that can happen! Can anyone provide a further explanation/material for why this is the case?
> The process was stuck in the kernel waiting for an IOCTL, so killing the process did nothing – even kill -9. It was going to stay that way until EBS was back to normal.
I did not know that can happen! Can anyone provide a further explanation/material for why this is the case?
Processes blocked in system calls can be unkillable. In the Linux kernel, what typically happens is that the process is blocked on a wait queue (what userspace usually call a "condition variable" or "monitor") in one of the wait_event() family of functions. Because the kernel doesn't implement any kind of exception handling, there is no way to automatically break out of this; you have to wait for some other task to call wake_up().
It's possible to wait in a "interruptible" way, and many subsystems do. But that's hard, because it means that the resources you would otherwise be responsible for clean up need to be detected and cleaned up by someone else (or alternatively that someone else needs to pick up and finish what you started, or you need to be able to roll back a partially completed action, etc...). Userspace usually ignoes this problem (because the task got killed, right?), but in the kernel a leak is a huge bug.
So the default is that waiting in the kernel means that the kernel can't allow the process to be terminated (or otherwise be delivered a signal that would invalidate the stack). And the side effect is that userspace sees an unkillable process in the "D" state.
It's possible to wait in a "interruptible" way, and many subsystems do. But that's hard, because it means that the resources you would otherwise be responsible for clean up need to be detected and cleaned up by someone else (or alternatively that someone else needs to pick up and finish what you started, or you need to be able to roll back a partially completed action, etc...). Userspace usually ignoes this problem (because the task got killed, right?), but in the kernel a leak is a huge bug.
So the default is that waiting in the kernel means that the kernel can't allow the process to be terminated (or otherwise be delivered a signal that would invalidate the stack). And the side effect is that userspace sees an unkillable process in the "D" state.
Three links for you:
https://jira.mongodb.org/browse/SERVER-6178
http://www.noah.org/wiki/Kill_-9_does_not_work
http://stackoverflow.com/questions/223644/what-is-an-uninter...
The first is the bug I reported to 10gen about their MongoDB setup. Contained in the bug report is the set of commands I used to reproduce the issue so that you can see it yourself.
The second and third links are about processes in uninterruptible sleep, which is the state my MongoDB processes were in. It should give more detail than I'm about to give.
Linux processes all have a state (running, sleeping, etc). One of those states is "Uninterruptible Sleep". Only the kernel can change a process's state. Uninterruptible sleep ("D" in `top`) is a state in which the process will receive no interrupts (signals). Since the `kill` command is just a glorified signal sender, it ceases to work for those processes.
https://jira.mongodb.org/browse/SERVER-6178
http://www.noah.org/wiki/Kill_-9_does_not_work
http://stackoverflow.com/questions/223644/what-is-an-uninter...
The first is the bug I reported to 10gen about their MongoDB setup. Contained in the bug report is the set of commands I used to reproduce the issue so that you can see it yourself.
The second and third links are about processes in uninterruptible sleep, which is the state my MongoDB processes were in. It should give more detail than I'm about to give.
Linux processes all have a state (running, sleeping, etc). One of those states is "Uninterruptible Sleep". Only the kernel can change a process's state. Uninterruptible sleep ("D" in `top`) is a state in which the process will receive no interrupts (signals). Since the `kill` command is just a glorified signal sender, it ceases to work for those processes.
IIRC is the only case where you can not kill a process under unices. It goes into the D state (uninterruptible sleep). Usually waiting for I/O is the reason for a process going into a D state. From my own experience, had this issue with faulty hardware and network filesystems. Here's something easy for reproducing a process going into uninterruptible sleep: GlusterFS 3.0.4 (Ubuntu 10.04 packages that version, a single machine setup is sufficient). A bug in Gluster turns into a deadlock and the process that triggers that bug does into the D state. Cause: using file locks without explicit unlock when releasing the file descriptor. The process can't be killed as it waits for the FS while the FS waits for the process to release the lock (which never happens). Had fun debugging that on a live deployment where each process got stuck one by one till the process pool was exhausted, while monit could not restart the daemon that managed the process pool.
I use the iptables reject trick all the time. It's particularly useful when you have a bunch of servers in a DNS round robin. When one of them adds the reject rule, clients start failing over to the next server in the DNS round robin essentially immediately.
How about using a proven high availability solution such as Corosync/Pacemaker instead of rewriting a limited implementation from scratch? http://corosync.org/
Up-voting you because, well, everyone always forgets about Corosync. About the only criticism one could have had about it was that when it only supported multicast you couldn't use it with EC2 and other providers that (for good reason) blocked multicast. It now can do UDPU so it works on EC2.
The other great thing about corosync+pacemaker is that it's super easy to write your own HA scripts. Need mongodb? Then write your own script(not sure why you'd use corosync for this, but you could...).
The other great thing about corosync+pacemaker is that it's super easy to write your own HA scripts. Need mongodb? Then write your own script(not sure why you'd use corosync for this, but you could...).
[deleted]
Maybe I'm missing something, but why not create a software RAID 1+0 array of disks (easily doable with EBS volumes and mdadm) instead of relying on a heartbeat-style solution as described in this post?
The world is full of wonderful alternatives like SoftLayer, for example (no affiliation). We've been running a very high traffic system there with individual box uptimes of 700 days before we had to reboot them during scheduled OS upgrade/maintenance.
The "AWS scaling stories" I hear at dev. meetups make absolutely no sense. People, AWS original sales pitch was to take care of scaling for you, not the other way around.
The developers obsession with AWS amazes me. It's like paying for a BMW and being proud of carrying around a self-made toolbox in case of an engine failure in the middle of nowhere, and everyone is proud to "know someone inside Amazon". Since when establishing a personal relationship with a car mechanic is seen as "cool"? Why not just pick a quality brand?