Reddit has now been down for 30+ hours(reddit.com)
reddit.com
Reddit has now been down for 30+ hours
http://www.reddit.com/?down
15 comments
Being a sys admin and having to plan disaster recovery I can understand this scenario. I've played it out in my mind, its been done to death on paper and we've tested it. My question is, why aren't reddit doing anything about it? After numerous outages over the past few months it seems reddit's answer to everything is, "It's not our fault, it's Amazon". In all honesty without turning this into an attack that is ridiculous. If my company were to suffer outages on a weekly basis, my boss and I would both be under the gun and seriously before it even got to that point we would be restructuring systems and shifting hosting platforms. Before anyone says to me, "hold on, you cant just go moving infrastructure". BS. When your business is the website and you are providing a service, availability is damn well key. I've moved datacenters more than once, I've delt with disaster recovery, I know how to do these things transparently and before you start thinking I'm arrogant well actually I only have 4 years of experience in the system administration game so I'd expect people with more talent than me to be able to get the job done pretty damn effortlessly. If our company suffered an outage of more than 3-4 hours we'd be switching to our DR plan end of story. That means switching dns and pushing to warm standby servers.
Engineers of reddit, take some initiative. Move off aws. I dont care how hard it is, look at the outages you've suffered. Look at the problems you've had. Havent you figured it out yet? Get some physical hardware with rackspace and sort it out.
Please understand this comment is not meant to attack anyone but I am frustrated with people shifting blame and not taking responsibility. If your down 30 hours then you have failed.
Engineers of reddit, take some initiative. Move off aws. I dont care how hard it is, look at the outages you've suffered. Look at the problems you've had. Havent you figured it out yet? Get some physical hardware with rackspace and sort it out.
Please understand this comment is not meant to attack anyone but I am frustrated with people shifting blame and not taking responsibility. If your down 30 hours then you have failed.
Reddit have real problems with staffing. They have said they are planning to do stuff like this, but their parent company Conde Nast won't let them hire anyone, for some corporate reason. They only have like one sysadmin IIRC.
Reddit has < 5 engineers working on it and in the last few months has been at 1 active engineer for a while. With 13mm unique visitors and > 1 billion page views monthly let's cut them some slack.
Let me ask you. How does a company get into that state? Say you had 5 engineers, 1 billion page views a month and ongoing issues. Wouldn't you dedicate some resources to that? When one engineer leaves, wouldn't you worry a little? And then another, and another? Would you sit and do nothing? These events did not occur in a day. They occurred over a lengthy period of time. I can sympathize with the lone engineer because I've been there. I still sleep with my laptop and phone next to my head. I know what the burn out feels like, so yes I feel for that engineer. But as a business letting it get to that point.
edit: JonnieCache mentioned Conde Naste will not let them hire more people. Obviously we know where the fault lies then.
edit: JonnieCache mentioned Conde Naste will not let them hire more people. Obviously we know where the fault lies then.
> JonnieCache mentioned Conde Naste will not let them hire more people.
Yeah, I imagine the situation is more complicated than that, but look into some of the reddit blogposts and some of the admins comments on here for more details. There was a lot of stuff in the big 'AWS is down' thread about it, something about CN giving them an unlimited operating budget but a tiny, non negotiable hiring budget. God knows.
Yeah, I imagine the situation is more complicated than that, but look into some of the reddit blogposts and some of the admins comments on here for more details. There was a lot of stuff in the big 'AWS is down' thread about it, something about CN giving them an unlimited operating budget but a tiny, non negotiable hiring budget. God knows.
Gist I got from the 'AWS is down' thread was they had maxed their operation budget and that was why they couldn't afford to use multiple regions. Reddit being very write heavy the data synchronization across regions would be too expensive in both bandwidth and manpower.
Conde Naste will not let them hire more people. Obviously we know where the fault lies then.
We don't know what's going on so I think it's a bit unfair to speculate.
But is there any evidence that Reddit is profitable? Would having a more stable platform honestly have much of an impact on that? Server issues aside, they seem to be doing pretty well in terms of page views and user acquisition.
We don't know what's going on so I think it's a bit unfair to speculate.
But is there any evidence that Reddit is profitable? Would having a more stable platform honestly have much of an impact on that? Server issues aside, they seem to be doing pretty well in terms of page views and user acquisition.
Given reddit’s content and its demographic appeal, I’d guess it’s got about as much revenue potential as 4chan, which is to say, barely any at all. Perhaps even less, considering Condé Nast won’t let reddit run ads from the porn sites and male enhancement product vendors that keep 4chan afloat. There’s a reason reddit has to resort to Flash games and lolcats as placeholders for its ad space.
If reddit became NSFW with ads you'd see a huge drop in traffic from people at work (that don't have access to Adblock on their computers) and a lot more people not disabling it for reddit.com at home.
Not only that, but if reddit started carrying those ads appropriate to its demographic, it would face a massive revolt from its own users. Redditors get irrationally angry at reddit whenever an ad targeted at them so much as plays sound, let alone insults their self-image as high-brow intellectuals who “deserve” premium ads on par with the sponsors of Condé Nast’s other properties. (Pretty funny stuff, for a site that had to petition Google to hide “jailbait” and “nsfw” from its top eight sitelinks.) But yes, that’s the dilemma.
[deleted]
Last I heard they had one full-time developer working on it, and if that is the case, I wouldn't be surprised if yesterday was their first day off in a awhile. Also, Quora was having the same difficulties and it mostly points to not doing what Twilio and a host of other architecture-aware teams did. Single points of failure will fail.
>If our company suffered an outage of more than 3-4 hours we'd be switching to our DR plan end of story.
I don't really see how you can have a functional DR plan for a site the size of Reddit with fewer than 10 people.
Of course, I've never been involved with keeping a site the size of Reddit operational, and I suspect the number of people required to do the kind of migration you're suggesting is an obvious choice would require Reddit to at least quadruple their staff.
I guess they could do it now, but odds are they would end up having to put Reddit in read-only mode for a week to do a proper migration with as tiny a staff as they have. And it's not clear that just throwing dedicated hardware at the problem would solve it.
I don't really see how you can have a functional DR plan for a site the size of Reddit with fewer than 10 people.
Of course, I've never been involved with keeping a site the size of Reddit operational, and I suspect the number of people required to do the kind of migration you're suggesting is an obvious choice would require Reddit to at least quadruple their staff.
I guess they could do it now, but odds are they would end up having to put Reddit in read-only mode for a week to do a proper migration with as tiny a staff as they have. And it's not clear that just throwing dedicated hardware at the problem would solve it.
>I don't really see how you can have a functional DR plan for a site the size of Reddit with fewer than 10 people.
You can definitely have a good plan, and be able to execute on that plan, with fewer people. I was lucky enough to work in a team that was running the search backend for a big internet company. Tens of thousands of machines, constant code, config and data pushes, configuration management, os images, kernel, etc. The team was responsible for developing and running the tools needed to keep the site up even when a whole datacenter lost power and you had to shift load from one coast to the other. The size of that team was much lower than 10 people.
Obviously Reddit will have to invest in both infrastructure and human resources to get there, but it can be done.
You can definitely have a good plan, and be able to execute on that plan, with fewer people. I was lucky enough to work in a team that was running the search backend for a big internet company. Tens of thousands of machines, constant code, config and data pushes, configuration management, os images, kernel, etc. The team was responsible for developing and running the tools needed to keep the site up even when a whole datacenter lost power and you had to shift load from one coast to the other. The size of that team was much lower than 10 people.
Obviously Reddit will have to invest in both infrastructure and human resources to get there, but it can be done.
A functional DR and usually the most optimal are the ones that require the least amount of work when disaster strikes. If our the datacenter in which our servers are hosted is destroyed I do not want to be figuring what hardware to restore databases on, where to deploy applications, what to use as load balancers, or what IPs to point my dns records to, what ad hoc config changes I had on a server somewhere. It should all have been preplanned, 80% of that should already have been done. We've got a 100mbit line running into the office. We've got staging servers, database servers and just crap loads of hardware thats used internally which would instantly become our server farm. Databases are being replicated as we speak, those that cannot are being incrementally backed up and restored. I've got a DR dns zone file ready to go, the ttl on dns records is 1800. If you had to do this stuff from scratch yes it would take time. But any credible business that runs online should have these things in place. If you're idea of a backup is your mirrored RAID I feel bad for you.
Does the size of their site matter? Perhaps. Does it require 10 engineers to get a DR strategy going and be able to implement it? No. If you clearly plan this ahead of time, if all the pieces are in place then no it will not take that many. If I was to take into account multiple pieces of infrastructure it could still be done. DR is not about getting the exact same infrastructure up. Its about keeping the business running, its about staying alive and in those situations usually you try the minimal that will get you there. Do I really need my back end reporting and emailing systems when shit hits the fan? No. Messages will queue up and when those systems come online later they'll take care of it.
Am I simplifying it? Perhaps so lets take an example.
In your current infrastructure you've got 100 servers, you got a couple load balancers, 40 webservers, 30-40 app servers and 20 database servers. Load balancers a dummy HA thing where something like haproxy, nginx, keepalived, lvs or something else runs, the second one is just the backup that takes over when the first dies. Those are static configs that dont take much to get up and running somewhere else. 40 webservers. So in my place of work each webserver is identical and any one of those can go down without affecting the other. Usually thats varnish, nginx, apache or something of the like in the front end maybe proxying off to rails apps, php or python. Each server has no dependency except on the database. Got some mid caching layer like memcached? Ok well you'd have one per webserver in a cluster and those things drop in and out with ease. App servers follow suit if you know how to build systems that are not tightly coupled and deal with asynchronous requests. Alot of it comes down to queues and message passing. Obviously there are some inter dependencies which you cant help and if you can solve some of it with intermediary haproxy balancing then great. So far I feel like all of these things can easily be setup on servers anywhere else and code can be frequently updated.
The real complexity is in data stores. I assume in companies with more than 20 machines they may have chosen multiple datastores for different purposes so it doesnt all live in one place. Yes that adds to the difficulty of things but you have to break things down and work on each one at a time. If you're smart you'll have used something that can be replicated, has multi master or is clustered in most cases you dont even have to worry about it because almost every popular datastore these days have those capabilities. Lets look at mysql. Most of the time people go with basic master slave replication and move to master with multiple slaves so you can offload reads. If you're replicating within the one place then theres no reason you cant replicate offsite to your warm standby. If you do maintain that offsite slave then you've got an upto the minute copy of your database in the event of catastrophic failure. Now thats one database down, what if your using redis, well that also has replication so I'm not seeing a problem there either. Mongodb, cassandra, oracle, it really doesnt matter what it is because you can maintain a copy.
At the end of the day its about planning, its about taking the time to already have 80% of the work done and getting the major pieces up and running. Theres no excuse. There really isnt. You think facebook's in one datacenter? Yea at one point they were but back then you really think they didnt have a backup plan? Ok they are big with lots of people but honestly I dont care if you've got 10 different datastores (dont do that) and each is half a terabyte on disk. I dont care what the complexity is, if you think its too complex to have a running backup thats your fault. Imagine google said, "eh yea the application is a bit complicated and we really cant have a backup somewhere else". What?!
Migrations, if you have no plan and are doing it from scratch yes that takes time. It took me a month from start to finish and thats only because I spent the first two trying to figure out how to get chef to automate most of it. But that was a migration and I had the luxury of time. If it was a DR scenario I'd have at max 8-10 hours before the MD would be yelling at me. Put all your databases on servers somewhere else, get the code up and running and have a strategy for switching.
Does the size of their site matter? Perhaps. Does it require 10 engineers to get a DR strategy going and be able to implement it? No. If you clearly plan this ahead of time, if all the pieces are in place then no it will not take that many. If I was to take into account multiple pieces of infrastructure it could still be done. DR is not about getting the exact same infrastructure up. Its about keeping the business running, its about staying alive and in those situations usually you try the minimal that will get you there. Do I really need my back end reporting and emailing systems when shit hits the fan? No. Messages will queue up and when those systems come online later they'll take care of it.
Am I simplifying it? Perhaps so lets take an example.
In your current infrastructure you've got 100 servers, you got a couple load balancers, 40 webservers, 30-40 app servers and 20 database servers. Load balancers a dummy HA thing where something like haproxy, nginx, keepalived, lvs or something else runs, the second one is just the backup that takes over when the first dies. Those are static configs that dont take much to get up and running somewhere else. 40 webservers. So in my place of work each webserver is identical and any one of those can go down without affecting the other. Usually thats varnish, nginx, apache or something of the like in the front end maybe proxying off to rails apps, php or python. Each server has no dependency except on the database. Got some mid caching layer like memcached? Ok well you'd have one per webserver in a cluster and those things drop in and out with ease. App servers follow suit if you know how to build systems that are not tightly coupled and deal with asynchronous requests. Alot of it comes down to queues and message passing. Obviously there are some inter dependencies which you cant help and if you can solve some of it with intermediary haproxy balancing then great. So far I feel like all of these things can easily be setup on servers anywhere else and code can be frequently updated.
The real complexity is in data stores. I assume in companies with more than 20 machines they may have chosen multiple datastores for different purposes so it doesnt all live in one place. Yes that adds to the difficulty of things but you have to break things down and work on each one at a time. If you're smart you'll have used something that can be replicated, has multi master or is clustered in most cases you dont even have to worry about it because almost every popular datastore these days have those capabilities. Lets look at mysql. Most of the time people go with basic master slave replication and move to master with multiple slaves so you can offload reads. If you're replicating within the one place then theres no reason you cant replicate offsite to your warm standby. If you do maintain that offsite slave then you've got an upto the minute copy of your database in the event of catastrophic failure. Now thats one database down, what if your using redis, well that also has replication so I'm not seeing a problem there either. Mongodb, cassandra, oracle, it really doesnt matter what it is because you can maintain a copy.
At the end of the day its about planning, its about taking the time to already have 80% of the work done and getting the major pieces up and running. Theres no excuse. There really isnt. You think facebook's in one datacenter? Yea at one point they were but back then you really think they didnt have a backup plan? Ok they are big with lots of people but honestly I dont care if you've got 10 different datastores (dont do that) and each is half a terabyte on disk. I dont care what the complexity is, if you think its too complex to have a running backup thats your fault. Imagine google said, "eh yea the application is a bit complicated and we really cant have a backup somewhere else". What?!
Migrations, if you have no plan and are doing it from scratch yes that takes time. It took me a month from start to finish and thats only because I spent the first two trying to figure out how to get chef to automate most of it. But that was a migration and I had the luxury of time. If it was a DR scenario I'd have at max 8-10 hours before the MD would be yelling at me. Put all your databases on servers somewhere else, get the code up and running and have a strategy for switching.
You can do high availability deployments using AWS. AWS offers different availability zones within a region, and multiple regions if you're extra paranoid. Unfortunately the way reddit is using the available services is not ideal. IIRC they are using cassandra backed by EBS which is not a best practice. It's much better to run cassandra on local storage and have a backup system that's uploading incremental and full snapshots of your data to S3.
I agree with your overall point that having a business continuity plan is required. They should be able to answer questions like how do we minimize impact when we lose one cassandra node or two, ebs gets corrupted, a whole avail-zone disappears, etc. I'm just saying that AWS provides the building blocks for you to do that.
I agree with your overall point that having a business continuity plan is required. They should be able to answer questions like how do we minimize impact when we lose one cassandra node or two, ebs gets corrupted, a whole avail-zone disappears, etc. I'm just saying that AWS provides the building blocks for you to do that.
IIRC from previous posts on HN, you should correct that to read "Engineer of Reddit".
They're an organization in flux, and when you're in that state, the priority is survival.
They're an organization in flux, and when you're in that state, the priority is survival.
Reddit's current technical staff is composed of a single engineer and 2 ops (maybe 3, but I believe it's 2).
They simply don't have the manpower to do that.
They simply don't have the manpower to do that.
[deleted]
Apparently all of Reddit's problem is either Amazon or Conde Nast.
They moved from their own infrastructure to AWS because they thought it would be more reliable and because they don't have much manpower.
I was wondering why I was logged in, able to comment and vote.
They're letting back in members slowly, with Reddit Gold members having a larger chance of being allowed in: http://www.reddit.com/r/reddit.com/comments/gv63r/nobody_can... . Be sure to read the comments by jedberg, one of the reddit staff.
Edit: Now the top of reddit says:
>UPDATE: We are slowly getting our capacity back and as such allowing a random subsets of redditors access to the site as we increase capacity. Please check back soon, as you may be able to log in if you are lucky. Thanks!
They're letting back in members slowly, with Reddit Gold members having a larger chance of being allowed in: http://www.reddit.com/r/reddit.com/comments/gv63r/nobody_can... . Be sure to read the comments by jedberg, one of the reddit staff.
Edit: Now the top of reddit says:
>UPDATE: We are slowly getting our capacity back and as such allowing a random subsets of redditors access to the site as we increase capacity. Please check back soon, as you may be able to log in if you are lucky. Thanks!
Maybe it's just me. I feel HN has become very sluggish this morning. Does that mean lots of redditors come over here?
Edit: Wierd, got downvoted. I do think this is related to the topic, so the failure of Amazon can have a domino effect, say, AWS fails -> Reddit fails -> HN sluggish/struggling -> ... Some times I really question the resilience of the social networks, as most of them may not be designed to handle 2x or 10x more load.
Edit: Wierd, got downvoted. I do think this is related to the topic, so the failure of Amazon can have a domino effect, say, AWS fails -> Reddit fails -> HN sluggish/struggling -> ... Some times I really question the resilience of the social networks, as most of them may not be designed to handle 2x or 10x more load.
It's amazing that AWS has been affected this long, or rather that their customers have been; here focusing on one big customer, Reddit. But actually what seems more relevant to me, vis-à-vis sites like Reddit are services such as Heroku which were or still are affected. Although I can appreciate that Reddit is an extremely popular site, at the end of the day I'm more concerned with the development services that were impacted.
Btw, has Amazon commented about potential irrecoverable/lost changes/damages, if any?
They are sort of limping back up now. Some users -- seems to be mostly gold users right now -- can get in.
It is reddit. Not experts-exchange. If it were "gold users can now login" that would be bad.
If they have to let users back in slowly (which they do), why shouldn't it be ones that contributed financially to the site?
If it has to be based on a metric, it should be based on the Karma. "Who contributed the most for the site."
They had a bad outage around a year ago which was the first time I saw the "emergency read-only" notice. Back then, the metric was "age of account". Even though by that metric, I would have been one of the first users back, I really don't mind that the Gold users are being let in on a "biased random" basis. As ceejayoz said, "why shouldn't it be the ones that contributed financially"?
The three metrics mentioned so far all have issues, but I think I like yours the least.
Gold - "Thanks for helping fund the site. Hopefully your dollars will let us hire folks that will ensure this doesn't happen again." Age - "Thanks for sticking by our little site through all of our outages. We know you've had to put up with a lot, but we appreciate it." Karma - "Thanks for all the cats. We love cats."
I realize that I'm lumping a lot of users who post great informative content in with the "better drink my own piss" image macro crowd, but karma just doesn't seem like the best metric here.
It's also harder to come up with a good threshold for karma. Where do you draw the line? 10k karma? 100k? Are we only including comment karma or submission karma? Both?
At least the Gold members have, in general, unambiguously contributed to the site. I don't have a problem using that as the primary metric here.
(I should note that I'm a reddit plebeian in this regard. I have no golden dog in this fight.)
The three metrics mentioned so far all have issues, but I think I like yours the least.
Gold - "Thanks for helping fund the site. Hopefully your dollars will let us hire folks that will ensure this doesn't happen again." Age - "Thanks for sticking by our little site through all of our outages. We know you've had to put up with a lot, but we appreciate it." Karma - "Thanks for all the cats. We love cats."
I realize that I'm lumping a lot of users who post great informative content in with the "better drink my own piss" image macro crowd, but karma just doesn't seem like the best metric here.
It's also harder to come up with a good threshold for karma. Where do you draw the line? 10k karma? 100k? Are we only including comment karma or submission karma? Both?
At least the Gold members have, in general, unambiguously contributed to the site. I don't have a problem using that as the primary metric here.
(I should note that I'm a reddit plebeian in this regard. I have no golden dog in this fight.)
This conversation is rather had there, at reddit. And since this morning, the response on the reddit itself is much on the lines of how I predicted it.
If I understand correctly, sites using AWS, and reddit.com for sure, are geared towards automatically starting and stopping EC2 instances as needed. If that is the case, why can't they just move everything to a different region, which hasn't been affected by this disaster?
Is there any proof that Conde Nast is trying to starve-off Reddit so they can maybe write it off on their taxes? They just don't seem to have a single ounce of pride in owning it.
It's amazing how much has been done with it's development given how little they have to work with.
It's amazing how much has been done with it's development given how little they have to work with.
I recently set up a website, and did consider AWS to host it. I decided not to, based on the problems Reddit had been having with AWS. I eventually got a VPS; looks like I made the right decision.
Counterpoint: We've been happily using AWS for several years now. We haven't had one second of downtime. We host across multiple availability zones, and have been pretty insulated. We are hosted in the eastern region (which was the one having issues yesterday), but managed to get through everything ok. We did lose some instances in one AZ, but failed back to our other instances without any incident.
ANY provider is going to have issues, the AWS actually gives you a bunch of nice options to easily handle those. I'm not saying that everyone should use Amazon, but don't take Reddits issues (I'm not familiar enough with their architecture to know where the fault really lies) to be the only data point for AWS reliability.
ANY provider is going to have issues, the AWS actually gives you a bunch of nice options to easily handle those. I'm not saying that everyone should use Amazon, but don't take Reddits issues (I'm not familiar enough with their architecture to know where the fault really lies) to be the only data point for AWS reliability.
If you do use AWS, however, it is vitally important to understand the tools they provide for redundancy, and their limitations. There are multiple availability zones for a reason, and although there was some multi-AZ downtime at the start, that recovered relatively quickly.
I'm kindof curious how they're going to deal with the self-serve ads that were supposed to run yesterday.
Just a straight refund, or...?
/me patiently waiting for it to come back up so that I can buy some for next week... :(
Just a straight refund, or...?
/me patiently waiting for it to come back up so that I can buy some for next week... :(
jedberg said yesterday he will add a day to the self-serve ads.
Sadly, my productivity has not improved one bit :(
I wonder what the users of Reddit are doing? going for a walk outside and enjoying the nature?
If anyone from reddit is reading this, I tried to sign up for Gold so I could log in, but I can't because signing up requires you to log in. I suspect I'm not the only person who would have happily bribed you to get to the head of the line. I think you guys left a lot of money on the table.
Seems to be like the community is well past the state of being angry when the site gets back up. They will be so glad that it is back at all. Need more donations Conde Nast?