Linode is down in Fremont, CA(forum.linode.com)
forum.linode.com
Linode is down in Fremont, CA
http://forum.linode.com/viewtopic.php?t=7080&postdays=0&postorder=asc&start=105
6 comments
I'm seriously considering moving my instance to another data-center. The Fremont facility has seen some disproportionate down-time these past few months.
The company that runs the data-center, Hurricane Electric, seems to be really dropping the ball when it comes to providing sufficient/consistent electricity.
The power system at the Fremont 1 location is really on it's last legs, and sucks badly. Fremont 2 is better. I'm not saying Fremont 2 is awesome, but it's not as ridiculously bad as 1.
They claimed they were going to buy new power equipment after the last outage; I'm actually not sure if this current outage is a power outage or what; it looks like all my servers at Fremont 1 are still up (and have been since the last outage, some 9 days ago) and the network hasn't been great, but it hasn't been down hard, either.
They claimed they were going to buy new power equipment after the last outage; I'm actually not sure if this current outage is a power outage or what; it looks like all my servers at Fremont 1 are still up (and have been since the last outage, some 9 days ago) and the network hasn't been great, but it hasn't been down hard, either.
The problem that we noticed with our servers was not that any of our servers went down. Linode's DNS servers for the data-center went down. Therefore, all of our servers did not know how to communicate with each other, and that took our stack down. As a preventative measure we are planning on installing a DNS cache on each server.
Hurricane Electric is a horrible ISP.
They offer bandwidth at very cheap prices but their hardware and connectivity is shit. The only reason why so many companies use HE is because they are up to 9 times cheaper than other ISPs. They got most of their peerings because they were an early IPv6 adopter.
I currently use Nearlyfreespeech.net and BuyVM/Frantech.ca for webhosting, and they both have/had problems with HE. NFSN uses HE currently only for IPv6 and BuyVM actually moved all their servers to Coresite. Prgmr.com also decided to move to Coresite, but I do not know if it was because due to problems at their Freemont location or not. (Maybe lsc can clarify this.) EDIT: As clarified by Luke, they did not move from Freemont (see his answer for more details).
They offer bandwidth at very cheap prices but their hardware and connectivity is shit. The only reason why so many companies use HE is because they are up to 9 times cheaper than other ISPs. They got most of their peerings because they were an early IPv6 adopter.
I currently use Nearlyfreespeech.net and BuyVM/Frantech.ca for webhosting, and they both have/had problems with HE. NFSN uses HE currently only for IPv6 and BuyVM actually moved all their servers to Coresite. Prgmr.com also decided to move to Coresite, but I do not know if it was because due to problems at their Freemont location or not. (Maybe lsc can clarify this.) EDIT: As clarified by Luke, they did not move from Freemont (see his answer for more details).
I haven't moved from he.net at all yet, and I'm not sure if I'm moving from he to coresite or to svtix. Corsite is not exactly a great data center, either; I ended up with space there mostly 'cause I'm and idiot and stayed on my upstream's IPs, which gave them a lot of leverage over me, which they, of course, used. So now I'm in he.net fremont (around 3.6kw usable, not entirely used) in SVTIX (around 7.2Kw usable, not entirely used) and in coresite (around 7.2kw, not entirely used)
The cost per watt is best at SVTIX, and SVTIX has by far the best power uptime record. (On the other hand, as the reseller who strongarmed me into mpt points out, SVTIX is run by a real-estate guy; it has no right to be as reliable as it is. But, it's a friendly real-estate guy who seems to be willing to listen to me, very non-corporate, so one option is for me to get more involved with managing SVTIX. Hell, I've started packing the BOC staff with my friends already, and my last reseller has burned his bridges with this guy, and burnt those bridges quite thoroughly, so I smell an opportunity for me.)
My K5 review of market post tower, a coresite data center: http://www.kuro5hin.org/story/2011/6/2/54516/27653
Please note, that review was written for a Kuro5hin audience; which is to say, it was written for the sort of people who would have been on 4chan 10 years ago if 4chan was a thing 10 years ago, so the tone is, uh, rather unprofessional. But the point stands, MPT is not a great data center. I mean, it's worlds ahead of he.net fremont 1, but that's not saying a lot.
Now, the big advantage of market post tower, the corsite location, is that it's the center of the world, bandwidth wise, so I should be able to do some peering and get more providers as I level up my networking knowledge, and I have a cheap gigabit line between market post tower and svtix.
As for he.net bandwidth, well, yeah. It's the cheap stuff. But really, it's about as reliable as most tier 1 ISPs; the thing of it is, he.net is a tier 2, and perfectly willing to pay for bandwidth if they can't peer, so peering spats (which are ridiculously common) are not a problem. Tier 1 ISPs? if they get in a peering spat (see the level3/cogent pissing match) you only have access to part of the internet.
Anyhow, in my position, it's absolutely retarded to only have one upstream. He.net and Cogent is a mix I'm looking at, and so long as I don't f*ck the BGP config, between the two of those, I should be both more reliable and cheaper than one of the higher end carriers.
The cost per watt is best at SVTIX, and SVTIX has by far the best power uptime record. (On the other hand, as the reseller who strongarmed me into mpt points out, SVTIX is run by a real-estate guy; it has no right to be as reliable as it is. But, it's a friendly real-estate guy who seems to be willing to listen to me, very non-corporate, so one option is for me to get more involved with managing SVTIX. Hell, I've started packing the BOC staff with my friends already, and my last reseller has burned his bridges with this guy, and burnt those bridges quite thoroughly, so I smell an opportunity for me.)
My K5 review of market post tower, a coresite data center: http://www.kuro5hin.org/story/2011/6/2/54516/27653
Please note, that review was written for a Kuro5hin audience; which is to say, it was written for the sort of people who would have been on 4chan 10 years ago if 4chan was a thing 10 years ago, so the tone is, uh, rather unprofessional. But the point stands, MPT is not a great data center. I mean, it's worlds ahead of he.net fremont 1, but that's not saying a lot.
Now, the big advantage of market post tower, the corsite location, is that it's the center of the world, bandwidth wise, so I should be able to do some peering and get more providers as I level up my networking knowledge, and I have a cheap gigabit line between market post tower and svtix.
As for he.net bandwidth, well, yeah. It's the cheap stuff. But really, it's about as reliable as most tier 1 ISPs; the thing of it is, he.net is a tier 2, and perfectly willing to pay for bandwidth if they can't peer, so peering spats (which are ridiculously common) are not a problem. Tier 1 ISPs? if they get in a peering spat (see the level3/cogent pissing match) you only have access to part of the internet.
Anyhow, in my position, it's absolutely retarded to only have one upstream. He.net and Cogent is a mix I'm looking at, and so long as I don't f*ck the BGP config, between the two of those, I should be both more reliable and cheaper than one of the higher end carriers.
Wow, thanks for the detailed answer.
I remembered from the time that I was using prgmr that you were moving to Coresite, but I must have gotten the details mixed up.
My problem with he.net is not about their peerings. They have a large number of peers, but their problem is that their network can not actually handle the bandwidth they are selling. Some time ago I had to upload about 120 GB of data to the Internet Archive and because of HE's cheap pricing the transfer of course got routed through them. Throughput during the day was averaging 100 KByte/s and only reached 2 MByte/s at night.
I remembered from the time that I was using prgmr that you were moving to Coresite, but I must have gotten the details mixed up.
My problem with he.net is not about their peerings. They have a large number of peers, but their problem is that their network can not actually handle the bandwidth they are selling. Some time ago I had to upload about 120 GB of data to the Internet Archive and because of HE's cheap pricing the transfer of course got routed through them. Throughput during the day was averaging 100 KByte/s and only reached 2 MByte/s at night.
are you sure it was you and not the Internet Archive? I haven't had trouble pushing my advertised capacity even during the day.
At the location i'm in, they do have all kinds of broadcast spam, though, which is at best, highly unprofessional.
We're slowly moving a bunch of our xen hosts from svtix to coresite, because we want this lump of servers to be on the same network, and because we're about to make a co-lo push at SVTIX. In the future, it's unclear if we will continue to buy more racks at coresite as we add more xen servers and keep svtix co-lo only, or if we will put another batch of xen servers at svtix.
At the location i'm in, they do have all kinds of broadcast spam, though, which is at best, highly unprofessional.
We're slowly moving a bunch of our xen hosts from svtix to coresite, because we want this lump of servers to be on the same network, and because we're about to make a co-lo push at SVTIX. In the future, it's unclear if we will continue to buy more racks at coresite as we add more xen servers and keep svtix co-lo only, or if we will put another batch of xen servers at svtix.
Yes, I confirmed it by transfering the data to a VPS hosted at Freemont (just as slow) and by transfering data from another host (that thankfully did not route via HE) to the Internet Archive.
The Internet Archive can be a bit slow sometimes, but given their budget I think they are doing fairly well.
The Internet Archive can be a bit slow sometimes, but given their budget I think they are doing fairly well.
I have already opened a ticket to move to their Dallas location. Many people have told me it is the best. Wish I knew this upon first signing up with them...
There's no defensible reason to host in a datacenter in an earthquake zone.
Seriously. Earthquakes are by far the biggest cause of outages I've seen, far more than power/UPS/cooling/network/disk issues. These silly fools!
They're rare events, but they're also inevitable, the disruption to service after a serious quake will be prolonged, and this factor is completely under our control when selecting a host, unlike the others you list. Most companies offer a choice of datacenters and it's pretty easy to pick one that is not sitting on a fault.
It's quite possible my sarcasm meter is broken, but really? He.net freemont goes down a few times a year. We get an earthquake big enough to actually break things around here once every 10 or 20 years.
Even the really good data centers have outages more often than that, and he.net is decidedly low end in price.
Even the really good data centers have outages more often than that, and he.net is decidedly low end in price.
your sarcasm meter is broken.
sorry; I plead Poe's law. I've seen a lot of people who wanted data centers in separate parts of the world before they even had so much as offline backups. I've had some customers who wanted multiple locations instead of RAID. "well, when the disk fails, we'll switch to the backup location" me: ??!?
Now this is how you do outage responses & notifications.
I'm not even a Linode customer but this kind of clarity and response is exactly what I want to see from my actual service providers.
I'm not even a Linode customer but this kind of clarity and response is exactly what I want to see from my actual service providers.
No, it's not. A fair bit about this sucked. I found out my linode was down from Cloudkick... tried to ssh, no go. Logged into Linode.. My dashboard says that everything's peachy and my Linode is running. I start freaking out thinking maybe I ran it out of disk space and it's thrashing so I start issuing reboot requests.. safe-mode-reboot requests, etc.. They stack up. Nothing's being processed. Can't console-in either.
Finally I contact support. They were VERY fast in answering, and when they did they pointed me at status post that said Freemont is kinda out...
WELL THEN WHY THE FUCK DOES MY DASHBOARD SAY MY INSTANCE IS RUNNING?
Lame. Oh well. The 45 minutes or whatever they were offline had another 10 minutes of strapped-CPU reboot thrashing to get through before it was back up.
Oh, and this is the third time this year at least Freemont's been down.
So yeah.. cheers for being nice, and whatever.. but I'm not suuper enamored with how much love HN gives folks for ridiculous fuck-ups followed by a "Hey guys, I'm really sorry but here's a long blog post about how our flux capacitor won't ever fail us again (until next month)" blog post.
[Edit: Yeah.. I know. Downvotes were expected. Sorry for the rant, but this little outage of theirs caused a pretty embarrassing situation with an important client. So I'm happy to trade some karma for the opportunity to say 'fuck' in all caps in a public place right now. Cheers.]
[Edit: Oh, the irony! I post a totally obnoxious comment who's valid-points-to-tantrum ratio is way off the tantrum scale... Downvotes immediately. Then I edit a little (heartfelt) apology into it, and upvotes away. This is exactly the sort of psychology that's got me at least a little bit confused and intrigued about HN, and that I was tantruming on about in the original comment.]
Finally I contact support. They were VERY fast in answering, and when they did they pointed me at status post that said Freemont is kinda out...
WELL THEN WHY THE FUCK DOES MY DASHBOARD SAY MY INSTANCE IS RUNNING?
Lame. Oh well. The 45 minutes or whatever they were offline had another 10 minutes of strapped-CPU reboot thrashing to get through before it was back up.
Oh, and this is the third time this year at least Freemont's been down.
So yeah.. cheers for being nice, and whatever.. but I'm not suuper enamored with how much love HN gives folks for ridiculous fuck-ups followed by a "Hey guys, I'm really sorry but here's a long blog post about how our flux capacitor won't ever fail us again (until next month)" blog post.
[Edit: Yeah.. I know. Downvotes were expected. Sorry for the rant, but this little outage of theirs caused a pretty embarrassing situation with an important client. So I'm happy to trade some karma for the opportunity to say 'fuck' in all caps in a public place right now. Cheers.]
[Edit: Oh, the irony! I post a totally obnoxious comment who's valid-points-to-tantrum ratio is way off the tantrum scale... Downvotes immediately. Then I edit a little (heartfelt) apology into it, and upvotes away. This is exactly the sort of psychology that's got me at least a little bit confused and intrigued about HN, and that I was tantruming on about in the original comment.]
You should have just jumped on IRC prior to panicking--you would have gotten an update instantaneously. Or better yet, you could idle a detached screen in there. When I had a Linode a few months back, there were constantly 500 rather helpful people in their channel.
Dude, I should have done LOTS of things (including not posting the above comment).. But the fact that their own site was saying "Everything's Clever!" really led to believe that I was the one that effed up... and I wasn't thinking clearly. My wife had literally delivered a big job to a client moments before so I was just in a hot panic to get it up.
Really, if there's a problem in Freemont.. and I'm on my dashboard for my Freemont Linode.. there ought to be a banner or something saying "Hey buddy, it's probly us this time."
Really, if there's a problem in Freemont.. and I'm on my dashboard for my Freemont Linode.. there ought to be a banner or something saying "Hey buddy, it's probly us this time."
Yeeah, that bit about where I said I wasn't a customer of them.
So, I'm mostly just talking about the responses I see on the status page and their follow up.
As a systems dude I prefer seeing a detailed response after the incident rather than excuses.
So, I'm mostly just talking about the responses I see on the status page and their follow up.
As a systems dude I prefer seeing a detailed response after the incident rather than excuses.
Your absolutely right about the dashboard, but does it give a notice if the whole data center is having problems instead of parts of it?
Just my luck. I had planned to move to Dallas after last week's outage, but one of my bloggers asked me to put it off for a while. Le sigh.
Interestingly enough my node is in the Fremont datacenter, and it hasn't been down since last week's power outage. I guess I'm one of the lucky people in the zone that didn't lose power.