DDOS on Namecheap Free DNS and Default DNS V2(status.namecheap.com)
status.namecheap.com
DDOS on Namecheap Free DNS and Default DNS V2
http://status.namecheap.com/?p=14846
118 comments
We are in the process of mitigating a large scale DDoS attack against our global DNS platform. We expect service to return to normal very shortly. Stay tuned and let me know if you have any questions. [email protected]
Just for future reference, it's usually considered a good idea to put your status page on completely independent infrastructure so that it stays up even when the rest of your stuff goes down. A status page that doesn't work during an outage isn't particularly useful.
Good point. The status page is on another cloud but since this is a DNS issue, the subdomain is down. In the future, we'll investigate running this page on a secondary DNS.
According to whois for namecheap.com, the DNS for that domain is hosted on dynect.net, and "host status.namecheap.com" resolves just fine (to 204.232.212.56), so it does not appear to be a DNS issue that is preventing your status page from working.
I stand corrected. It's on Rackspace, which apparently is also experiencing issues.
Rackspace helped us get our status page back online: http://status.namecheap.com/
Perhaps also investigate allowing customers to slave their own secondary DNS, too? (That is, allow AXFRs.)
This has been a feature requested for, as far as I can tell from the support forums, four years now. It should be possible to make this allowed/disallowed per-zone, and as far as I'm concerned, I don't care who is able to download my tiny zone file. It would allow me to add more diversified DNS servers in the face of things like a DoS against Namecheap.
This has been a feature requested for, as far as I can tell from the support forums, four years now. It should be possible to make this allowed/disallowed per-zone, and as far as I'm concerned, I don't care who is able to download my tiny zone file. It would allow me to add more diversified DNS servers in the face of things like a DoS against Namecheap.
Posting up-to-the-minute updates on Twitter is also a good idea. Lots of tweets come back from a search for "Namecheap" & you want to be sure you're a part of that conversation!
They've been posting updates 30 minutes before your post buddy, and still are now. https://twitter.com/Namecheap
Yes, we are actively posting updates on Twitter.
If the hackers really want to take you down, adding another server in isn't very hard...
If you're as big as NameCheap, you can afford to pay CloudFlare or somebody like it to protect your status page.
Cloudflare has also been nailed the last 3 weeks in a row causing outages. Just search Twitter for cloudflare DDOS or NTP or etc. etc. etc. At the end of the day there are currently so many slave machines out there we are all vulnerable. It's just the nature of things. At least outages are only temporary. It's much better than the early days were we'd be down for 12 hours at a time. https://twitter.com/search?q=cloudflare%20attack&src=typd
I suspect the hackers are targeting one of namecheap's customers, not namecheap directly. Because that's usually the case, a good approach is to not give all customers the exact same nameservers.
I agree. We left Network Solutions b/c they were hit 3 times in less than a year. Maybe the website targets moved to Namecheap, so they were targeted also?
Like!
Not having separate status hosting is liken not salting passwords. As a NC user, I may leave just for this reason
WOOH the Akamai realtime attack visualization is completely red at the moment:
http://www.akamai.com/html/technology/dataviz1.html
Most DDOS attacks against a company like yours are actually attacks against a specific customer. If:
a) you had a pool of DNS server names, say 20, all with unrelated hostnames
b) you assigned 2 to each customer, randomly, when they configured a domain to use your servers.
Then, a DDOS attack would impact 10% of your customers instead of 100%. (Assuming other practices, like null routing the target until resolved)
a) you had a pool of DNS server names, say 20, all with unrelated hostnames
b) you assigned 2 to each customer, randomly, when they configured a domain to use your servers.
Then, a DDOS attack would impact 10% of your customers instead of 100%. (Assuming other practices, like null routing the target until resolved)
Yes, this. Hopefully they take heed after this painful experience.
Yes please, thank you :) All my domains are down now but I understand how shitty (the panic!) it feels to have things going down. I'll be checking this.
I'd also like to add that we have redunancy on our DNS V1. I advise switching over to this, in the meantime. Please find the tutorial here: https://www.namecheap.com/support/knowledgebase/article.aspx...
I tried this and now for some reason my apache server is returning the default page. also I think it wiped out my mailserver settings -- my mail is hosted on namecheap. i guess i gotta wait till everything is fixed.
I had a couple of domains to try this on. One domain switched over within a minute. The other has been almost an hour. I'm assuming the TLD makes a difference?
.com was < 1 minute, .cm is > 55 minutes.
.com was < 1 minute, .cm is > 55 minutes.
ETA how long it will take for the transfer to be completed? I know that editing host records usually is instantaneous (unlike other providers), but we're talking about changing the DNS servers here.
Looks to be up around five to ten minutes after executing the change. Not too terrible.
Thanks! I just did the switch and the website was up in few minutes.
Was going to switch to route 53 while you guys were down and switch back, but the page says it'll take a day...Might as well just wait at that point. I know it's panic mode over there, but some kind of failover record would be awesome for when this happens in the future(or an option for one).
Everything related to DNS says it'll take a day but that's "worst case"... I've successfully transferred DNS for 5 Namecheap-registered domains to Route 53 since this all started a couple hours ago, and they're now up & running smoothly.
Their "update DNS server" page was acting a bit wonky, kept saying some of my nameservers were invalid when they weren't, but I eventually got them all switched.
This isn't a dig against Namecheap, it sounds like this attack is pretty bad, but for important domains Route53 just seems like a much better setup (geographically dispersed, different nameservers for each hosted zone, etc).
Their "update DNS server" page was acting a bit wonky, kept saying some of my nameservers were invalid when they weren't, but I eventually got them all switched.
This isn't a dig against Namecheap, it sounds like this attack is pretty bad, but for important domains Route53 just seems like a much better setup (geographically dispersed, different nameservers for each hosted zone, etc).
Yeah, Route53 isn't as feature rich though. Also I have the same problem, keeps giving me errors about bad nameservers. I guess I'll keep trying.
[deleted]
Hi, I work on Route 53, can you clarify; are you seeing a problem from your registrar when you try to move to Route 53? We're also always keen to hear about features customers would like.
Secondary DNS from a normal bind nameserver.
I second this. I'll add that Route 53 would need to respond to notifies for this to be useful.
It's definitely on our list, but in the meantime there is https://code.google.com/p/route53d/ which will let you translate an AXFR/IXFR from a bind server directly into Route 53 authenticated calls.
I took your advice and transferred a domain to DNSv1. That domain is still not back online, but all the domains I left as DNSv2 have come back!
You guys should add an option for us to put in an optional backup DNS record in the settings.
The quick tut on switching to DNSv1 is nice however the option doesnt exist in my account
Thanks for posting here!
Kudos for using a HN page to stay in touch
Ho quickly will the switch to v1 execute?
It generally takes less than an hour.
Does it take the same amount of time to complete that switching to another provider's DNS server takes?
Many of us already switched to other DNS servers during the panic, and we'd like to know whether you'd suggest we reverse that change in order to jump to v1 instead for a quicker result. Thank you.
Many of us already switched to other DNS servers during the panic, and we'd like to know whether you'd suggest we reverse that change in order to jump to v1 instead for a quicker result. Thank you.
hi already have OK's, but the DNS are all using the same IP :( bad sign?
Thanks for the update, guys.
Good luck guys!
Down again
My domain down again
You need to provide some gift to your customers for this downtime. I am using NC for 10 years, every time on bad issues I continue to use. But this outage very bad, I lost money...
DDOS attacks are not up to Namecheap. You want money, go talk to the people doing it. These are very hard to prevent.
[deleted](2)
They could better protect against it by not assigning the same list of DNS servers to every customer.
Are you idiot or kid? We are doing professional business and we were using namecheap DNS with registered domains (not free dns). The main point is protection, they provide this service FREE (with or without domain) yeah good but where is protection where is responsibility?
I just suggested something, if money important for me, I didn't move all of my domains from namecheap to another provider :)
I just suggested something, if money important for me, I didn't move all of my domains from namecheap to another provider :)
That protection and responsibility is defined in your contract or registration agreement with NameCheap. You can consider DNS or DDoS protection NameCheap's responsibility, and they'll decide -- like any business -- which one they can afford.
While offering some gift or compensation for ANY negative incident -- especially one outside anyone's control -- is nice, that's not something everyone can afford under ANY circumstance.
While offering some gift or compensation for ANY negative incident -- especially one outside anyone's control -- is nice, that's not something everyone can afford under ANY circumstance.
[deleted]
Weird, I haven't researched DNS as well as I should have. I always lived under the impression that there was this extensive DNS cache network where intermediaries responded to queries with cached results from root DNS servers.
Instead, the second that this DDos hits is the second we have websites stopping working.
How is it that in this day and age we can't have distributed caches of DNS entries at our providers of full dns databases. I mean there can't be more than like a few billion dns entries in the world total, which fits easily in a modern desktop computers RAM.
If that is an underestimate, I can't believe a single modern server wouldn't be able to mirror the world's DNS queries for at least a providers worth of users.
Instead, the second that this DDos hits is the second we have websites stopping working.
How is it that in this day and age we can't have distributed caches of DNS entries at our providers of full dns databases. I mean there can't be more than like a few billion dns entries in the world total, which fits easily in a modern desktop computers RAM.
If that is an underestimate, I can't believe a single modern server wouldn't be able to mirror the world's DNS queries for at least a providers worth of users.
Depends on the TTL (time to live) settings for the DNS entries, doesn't it?
Yes, perhaps there lies our folly. It's the choice between being flexible in ability to move our servers really quick, or being tolerant of DNS servers going down.
I sort of hoped that a DNS client would just use an expired DNS result in case the servers would not respond, but perhaps that is naieve/dumb.
I sort of hoped that a DNS client would just use an expired DNS result in case the servers would not respond, but perhaps that is naieve/dumb.
> I sort of hoped that a DNS client would just use an expired DNS result in case the servers would not respond
This would break the whole concept of TTLs.
> but perhaps that is naieve/dumb.
Not at all. Hard problems are hard to solve.
This would break the whole concept of TTLs.
> but perhaps that is naieve/dumb.
Not at all. Hard problems are hard to solve.
How would one add (say) AWS Route53 as a secondary DNS?
I assume you'd make sure the DNS records are the same in both DNS portals; and then add Route53 as 3rd & 4th nameservers with the first and second still being Namecheap?
I assume you'd make sure the DNS records are the same in both DNS portals; and then add Route53 as 3rd & 4th nameservers with the first and second still being Namecheap?
It seems like secondary DNS is not supported, but you can change your primary DNS with the "Transfer DNS to Webhost" option.
Yes, or just switch completely to Route53.
If your site is down and you are on v2, Switch to v1. It only takes a minute and it works.
Yeah - at least right now this worked for me. Thank you for the tip!
I switched and it's the same, still down.
At first I thought the same. I went in to edit records, and I hit save. I checked again and it was working. I assumed there was just a delay of a minute or so and that clicking save was a coincidence.
My domain is on v1 now and its still working.
My domain is on v1 now and its still working.
Well even if you switch to a different (working) nameserver, the old nameserver will still be cached all over the place so it will still appear down for many people.
I recently switched most of my domains to DNSMadeEasy because they are constantly in the top for speed[1], provide a top tier anycast network and for what you get are a great value.
If you want speed and readability I suggest switching to a paid DNS provider.
1: http://www.solvedns.com/dns-comparison/2014/01
BTW I'm not in any way affiliated, just like the service.
If you want speed and readability I suggest switching to a paid DNS provider.
1: http://www.solvedns.com/dns-comparison/2014/01
BTW I'm not in any way affiliated, just like the service.
Best of luck to their support team. Outages can make tech support's life miserable. If you call in, just remember the person on the other side of the phone has likely been yelled at all morning for something that wasn't their fault. Totally reasonable to be upset at the situation, just don't take it out on the tech you're talking to!
Job security ;).. j/k! I imagine it must not be fun
[deleted]
May this have been a problem lasting for a week?
I am monitoring a few servers with DNS records. And the last week I have found all the servers unresponsive (by DNS, not tried directly) from time to time. And after an extensive amount of troubleshooting I am unable to find a problem.
I am monitoring a few servers with DNS records. And the last week I have found all the servers unresponsive (by DNS, not tried directly) from time to time. And after an extensive amount of troubleshooting I am unable to find a problem.
If you're affected, you can switch your domains to their DNSv1. Seems pretty quick for most people.
Via https://news.ycombinator.com/user?id=edwhitesell
Via https://news.ycombinator.com/user?id=edwhitesell
I don't know how this website works, but I can't see the latest posts at the top of the page! I'm looking for the latest info on the issue. Are you up and running? Should I move back to v2? Thanks
Is there any point in freaking out or do we just have to wait this one out?
If it's absolutely critical that your users get service right now, it might be a good idea to at least prepare a migration to other DNS servers, like perhaps those of Linode. If the situation doesn't improve within an hour or so, it might be that they don't have a good way to deal with it, and the outtage might take long, depending on the depth of the DDoSers pockets.
Any good suggestions for alternative DNS providers?
Instead of putting all of your eggs in (yet another) one basket, for a site with critical availability requirements I would distribute DNS over multiple providers. If any one provider goes down it is less likely to hurt overall site availability.
Yes, multiple DNS providers should be the way to go.
DNSMadeEasy, I just posted about them in this thread: https://news.ycombinator.com/item?id=7275538
I moved most of my stuff to Route53 awhile back and for the most part I don't regret it.
Yes, Route 53 has been great for me as well. I have my crucial DNS zones hosted there. But all the other stuff was on Namecheap's name servers.
I'll probably continue with that setup, but just make sure my TTL is set fairly high.
I'll probably continue with that setup, but just make sure my TTL is set fairly high.
Pricing doesn't seem too bad. $0.50 for a zone (domain) and $0.50 for a billion queries: http://aws.amazon.com/route53/pricing/
Correction on the queries pricing:
$0.500 per million queries – first 1 Billion queries / month
$0.250 per million queries – over 1 Billion queries / month
$0.500 per million queries – first 1 Billion queries / month
$0.250 per million queries – over 1 Billion queries / month
Yep. I had been hesitating on moving more domains over to Route53 because I thought the per-zone pricing was fixed at $0.50 per domain, but it actually scales really well. I'm going to move the rest over to Route53 as soon as I have a chance.
We're using Route 53 in production to handle hundreds of millions of queries per month. We're happy with it, and the price is reasonable. Also, service checks with failover is heavenly.
We looked at Route 53, but would that make a difference today if your domain is registered at Namecheap?
If Namecheap is down, it does not get redirected.
If Namecheap is down, it does not get redirected.
If you don't use namecheap's dns servers, dns requests for your domain do not go to namecheap for redirection. They don't go to namecheap at all. Your specified dns servers get registered with the top level domain servers.
If you use Route 53, your zone file is still at Namecheap (registrar). Doesn't that still make Namecheap a failure point?
No. The zone file lives on the nameserver (DNS server) which Route 53 provides. Namescheap registers a list of nameservers for your domain with the TLD.
An over simplified example for looking up "news.ycombinator.com":
1. First query the TLD nameserver for all ".com" domains asking for the authoritative nameserver(s) for "ycombinator.com".
2. Next query that nameserver for "news.ycombinator.com".
An over simplified example for looking up "news.ycombinator.com":
1. First query the TLD nameserver for all ".com" domains asking for the authoritative nameserver(s) for "ycombinator.com".
2. Next query that nameserver for "news.ycombinator.com".
But, the first entry point is namecheap. The domain is at Namecheap, so it will not find the zone file at Route 53, if Namecheap does not send it there. Right?
No, registrar's are your portal to updating data in TLDs, but those TLDs are each operated separately. .com and .net are operated by Verisign, for example.
[deleted]
I've considered doing this also!
We've been stoked with DNSimple.
This is so embarrassing for me. We just put out our school computer group's website up and boom - murphys law.
That would explain why I'm getting a massive amount of tickets from my clients why their site is down.
Down for me as well. Lost access to absolutely all of my company's services. TTL 60...
yup, our app is down : (