Digital Ocean was down on the West Coast(digitaloceanstatus.com)
digitaloceanstatus.com
Digital Ocean was down on the West Coast
http://www.digitaloceanstatus.com/
8 comments
More-expensive providers go down too. I would never put all my hosting eggs in 1 provider basket. That is asking for trouble. Even big-names like AWS can go down.
True, but more expensive providers have better SLAs. Take Rackspace, for instance. You pay A LOT for their services, but they have more incentive not to go down. They credit you 5% of your monthly fee for each 30 minutes of network downtime. So it isn't just their reputation on the line when an outage occurs. It's quite a bit of money, too.
Source: http://www.rackspace.com/managed_hosting/support/serviceleve...
Source: http://www.rackspace.com/managed_hosting/support/serviceleve...
Sorry for the above, I meant $0.0002 back per minute of outage, not hour. That would be $.0139 per hour back.
I wonder if the people paying $40,$80,$160.... $960/month were affected? I'm thinking yes.
True. But even if you were paying $1,000 a month, a 1 hour outage only gets you a $1.39 credit on your bill.
And it's not a matter of how much. It's a matter of how much for what you get, versus how much for what you get from elsewhere. A product that is significantly less expensive simply can't have the same level of hardware and technical support. This also shows in the SLA. Smaller, less expensive providers will credit you for lost time. Larger, more expensive providers will credit you for a % of your monthly bill per hour as a penalty for themselves.
And it's not a matter of how much. It's a matter of how much for what you get, versus how much for what you get from elsewhere. A product that is significantly less expensive simply can't have the same level of hardware and technical support. This also shows in the SLA. Smaller, less expensive providers will credit you for lost time. Larger, more expensive providers will credit you for a % of your monthly bill per hour as a penalty for themselves.
Customers of all sizes were affected. We are very sorry for this and are working extremely hard to ensure that our network is completely stable moving forward so that all customers experience 100% uptime.
So far this week I've watched Network Solutions get hosed down, Apple stay down, saw Ubuntu mention their forum user base has been exposed, my normally reliable internet connection has flaked out several times, and DigitalOcean has gone titsup for the moment because of a peering issue. It makes me wonder. It's my hope its not the new normal.
It's basically normal unless you are paying for a guaranteed SLA. Small hosting providers either have none or have one that just credits you for the time you were down, which is usually not much (an hour is 1/720th of your monthly fee).
Most technology is simply unreliable. Unless you pay a lot more for redundant systems. My internet provider here in NYC, Time Warner Cable, has had 3 outages in 3 weeks, two being a full day in my neighborhood. And that's on a $100/month 50 MB/s down and 5 MB/s up connection, the most expensive one they offer to the home. Unfortunately, like a lot of the US, Time Warner's infrastructure is horribly outdated, so these issues occur regularly. And, as with much of the US, we don't have any other options. Neither of the other cable providers service us. Verizon's FiOS rollout was abandoned. We can get DSL, but it's horribly slow (under 1MB/s up) and is notoriously reliable (when I tried it years back, it was down 8 days out of my 30 day trial period because a Verizon tech 'plugged something in wrong in our street's box'). So, I have to pay for a Verizon hotspot as a backup so I can 'reliably' work from home. Of course, the 1st of the 3 most recent outages Time Warner had, Verizon's LTE network went down at the same time.
So, yeah, that's the 'normal' of technology in the largest city in the United States.
Most technology is simply unreliable. Unless you pay a lot more for redundant systems. My internet provider here in NYC, Time Warner Cable, has had 3 outages in 3 weeks, two being a full day in my neighborhood. And that's on a $100/month 50 MB/s down and 5 MB/s up connection, the most expensive one they offer to the home. Unfortunately, like a lot of the US, Time Warner's infrastructure is horribly outdated, so these issues occur regularly. And, as with much of the US, we don't have any other options. Neither of the other cable providers service us. Verizon's FiOS rollout was abandoned. We can get DSL, but it's horribly slow (under 1MB/s up) and is notoriously reliable (when I tried it years back, it was down 8 days out of my 30 day trial period because a Verizon tech 'plugged something in wrong in our street's box'). So, I have to pay for a Verizon hotspot as a backup so I can 'reliably' work from home. Of course, the 1st of the 3 most recent outages Time Warner had, Verizon's LTE network went down at the same time.
So, yeah, that's the 'normal' of technology in the largest city in the United States.
Hi Chuck - I'm one of the co-founders. It is definitely not the new normal. We are working diligently to ensure that all networking issues will be permanently fixed so that all customers will experience 100% uptime. We will not stop improving our network till this happens. I promise you.
This is what is displayed on the top of the website in an alert:
-----------------
Peering Outage in SFO1 Sun Jul 21 19:15:00 2013
Upon further investigation is also appears nlayer might have a larger problem on the west coast. We have taken the circuit out of service and have a ticket opened with them to investigate. Sun Jul 21 19:50:34 2013
We saw some brief outages for one of our peers in SFO1.
While BGP converged on the backbones to reroute the inbound traffic from nlayer to our other peers, you might have experienced a short outage. This is standard bgp behavior and how the internet reroutes around outages.
Sun Jul 21 19:20:01 2013
--------------------
It's not clear to me whether this is telling me that there was an outage or that there is an outage. Regardless, my servers are inaccessible.
-----------------
Peering Outage in SFO1 Sun Jul 21 19:15:00 2013
Upon further investigation is also appears nlayer might have a larger problem on the west coast. We have taken the circuit out of service and have a ticket opened with them to investigate. Sun Jul 21 19:50:34 2013
We saw some brief outages for one of our peers in SFO1.
While BGP converged on the backbones to reroute the inbound traffic from nlayer to our other peers, you might have experienced a short outage. This is standard bgp behavior and how the internet reroutes around outages.
Sun Jul 21 19:20:01 2013
--------------------
It's not clear to me whether this is telling me that there was an outage or that there is an outage. Regardless, my servers are inaccessible.
Please open a ticket and request an SLA credit. This was an unexpected outage. Very sorry this happened.
Thanks. Shortly after I posted this, SSH access was restored and I am back online.
They are back up for me now.
I have not been at all happy with the support I get from DO.
I think they only employ BOFHs (http://en.wikipedia.org/wiki/Bastard_Operator_From_Hell)
I have not been at all happy with the support I get from DO.
I think they only employ BOFHs (http://en.wikipedia.org/wiki/Bastard_Operator_From_Hell)
Well, the one time I contacted their support I got a helpful reply within 3 minutes, so YMMV.
It wasn't so much about the response time, but more like they just did not care about my request.
There Debian image is not a standard image and I wanted to know what they had done so that I did not get caught by another 'difference' and they said that they did not have any documentation on what they had changed but I was free to change it to be however I wanted.
The problem with this is that if you use Vagrant or any other automated deployment anywhere else, you will either have to make changes to your build script to account for the changes they made or make your images to be like theirs are.
The entire process is rather busted - they should just use standard images or let me upload my own.
Their response was not at all feeling or understanding.
I have 1 $40 instance and will use it until all my credit is done but will not be returning.
There Debian image is not a standard image and I wanted to know what they had done so that I did not get caught by another 'difference' and they said that they did not have any documentation on what they had changed but I was free to change it to be however I wanted.
The problem with this is that if you use Vagrant or any other automated deployment anywhere else, you will either have to make changes to your build script to account for the changes they made or make your images to be like theirs are.
The entire process is rather busted - they should just use standard images or let me upload my own.
Their response was not at all feeling or understanding.
I have 1 $40 instance and will use it until all my credit is done but will not be returning.
We're sorry you feel this way and totally respect your opinion. Any additional feedback on why would be helpful. Thanks!
I would gladly get on a phone call with you. My contact information is attached to droplet id: 281363
Holy crap, who decided that light grey on white was a good color scheme for the timestamps? I know it's just a trendy design blunder but when I've nothing else to do but sit here and refresh the status page it feels like insult added to injury.
Zero timezone customization as well - In times of stress, who want to have to calculate UTC... really.
This is so frustrating. Linode charges 4 times more, but at least they're not down every few days.
Pro Tip: These two things often have a lot to do with each other.
We know it is very frustrating and completely agree with you. We are working hard so this won't happen again.
their site makes it sounds like bgp is rerouting, but i'm still down
Agree, seems a bit disingenuous, my Pingdom is showing 28 minutes of downtime and growing.
http://pingdom.commando.io/894360
http://pingdom.commando.io/894360
This is not good enough.
We completely agree. Please open a ticket to receive an SLA credit.
For folks who complain that Digital Ocean is unreliable compared to other more-expensive providers, did you really expect $5-10/mo hosting to be reliable? It isn't supposed to be rock-solid reliable. It's cheap hosting for messing about with.