Network Solutions' DNS was down(networksolutions.com)
networksolutions.com
Network Solutions' DNS was down
http://www.networksolutions.com
18 comments
This is one of the reasons why I self-host DNS. Even with tons of users, the resources it takes to serve DNS requests pale compared to what you have to put behind your application servers.
Of course if you are using a CDN to provide your users with better locality, you might want to look into a service that provides localized DNS distribution, but the inherent caching feature of the DNS protocol might make that an unneeded additional burden (I remember a very old Stack Exchange podcast episode where they were talking about this, coming to the conclusion that self-hosted DNS and dns-internal caching is good enough for them).
When you self-host your DNS, you ensure that no provider can suddenly redirect to an ad-filled parking pages if users mistype hostnames and you can make sure that you can fix DNS when it's down. It's also much easier to transfer domains between registrars because there's no need to export and import DNS config - instead, your server just keeps serving the exact same data.
Finally, this allows you to keep the DNS config together with all other configuration files in puppet/chef/git/whatever you use. which further helps future deployments and/or configuration changes.
Of course if you are using a CDN to provide your users with better locality, you might want to look into a service that provides localized DNS distribution, but the inherent caching feature of the DNS protocol might make that an unneeded additional burden (I remember a very old Stack Exchange podcast episode where they were talking about this, coming to the conclusion that self-hosted DNS and dns-internal caching is good enough for them).
When you self-host your DNS, you ensure that no provider can suddenly redirect to an ad-filled parking pages if users mistype hostnames and you can make sure that you can fix DNS when it's down. It's also much easier to transfer domains between registrars because there's no need to export and import DNS config - instead, your server just keeps serving the exact same data.
Finally, this allows you to keep the DNS config together with all other configuration files in puppet/chef/git/whatever you use. which further helps future deployments and/or configuration changes.
You'd think that it would be a good idea... until someone decides YOU are going to be the target of a DDoS attack. I speak from experience. I have many (at least 15) years of experience dealing with DNS and hosting of it, yet when a bot-net decided to attempt to use our DNS servers for a reflection attack. Obviously the servers were setup to not allow this, but the sheer volume of the traffic making attempts was still enough to saturate the connection with our provider at the time.
This is only one of many possible scenarios with self-hosted DNS. Essentially there are services out there (generally Anycast are best) that do only DNS and have had very high up-time. I'd suggest if you are anything over a 'small' target to look into these services. Sadly it's one of the few things I basically suggest not self-hosting at this point simply due to the unrealistic requirements to scale it yourself under attack scenarios.
This is only one of many possible scenarios with self-hosted DNS. Essentially there are services out there (generally Anycast are best) that do only DNS and have had very high up-time. I'd suggest if you are anything over a 'small' target to look into these services. Sadly it's one of the few things I basically suggest not self-hosting at this point simply due to the unrealistic requirements to scale it yourself under attack scenarios.
To Rhizomes point: Yes, it's relatively easy to re-point them to somewhere else.. and if your business can tolerate the interruption that's fine.
Those changes, though, take time to cache out, and depending on your registrar's procedures and whatnot, you could be looking at hours or a couple of days to get things to stabilize back to normal; not to mention finding a new provider and getting things set up. You'll need ot make sure you have the zone data on hand ready to get set up with a new provider, and so on. You could have a procedure in place to do this, sure, with everything tested and validated, but at that point it's much easier to just have a DNS provider of the proper class who can handle these things for you. And keep in mind, if it's your domain they are trying to shut down, and not your DNS provider specifically, the attack will very likely just follow the change, leaving you in quite a mess.
So the short answer: self-host if it's not revenue impacting. If there is money at stake, spend a little bit for a top-notch provider.
Those changes, though, take time to cache out, and depending on your registrar's procedures and whatnot, you could be looking at hours or a couple of days to get things to stabilize back to normal; not to mention finding a new provider and getting things set up. You'll need ot make sure you have the zone data on hand ready to get set up with a new provider, and so on. You could have a procedure in place to do this, sure, with everything tested and validated, but at that point it's much easier to just have a DNS provider of the proper class who can handle these things for you. And keep in mind, if it's your domain they are trying to shut down, and not your DNS provider specifically, the attack will very likely just follow the change, leaving you in quite a mess.
So the short answer: self-host if it's not revenue impacting. If there is money at stake, spend a little bit for a top-notch provider.
This has been my experience as well. Stupid bots being denied recursive requests. I modified my fail2ban script to autoban them which helped considerably.
Well, fail2ban can help, but that is only going to be generally quick enough to keep up with the initial ramp-up of the attack I'd expect. Once the full attack gets going you're probably going to be seeing requests from thousands of hosts at any given time. I haven't specifically tested under that scenario with fail2ban, but I imagine that it would have trouble keeping up with such a flow.
My experience was that fail2ban mitigated it pretty much completely for my small server (which no doubt was only allocated a small number of bots to be attacked with). It took about an hour to build up a list of about 16,000 machines and then basically things were fine again.
This DDoS argument really is a premature optimization, and it's very simple to implement an external primary/secondary if the problem ever does occur.
Read comment up-stream. I disagree with this standpoint and I'm sure can find you stories out there beyond mine to back up what I'm suggesting. DNS hosting as well is so inexpensive unless you're in the hundreds of millions of queries that really I see no reason to not look to hosted DNS solutions first. Even one of the premier DNS providers out there is only like $29 for a small-business plan.
Essentially, unless your website is not revenue generating (or is below the cost to make this feasible), you really should look at moving from self-hosted DNS.
Essentially, unless your website is not revenue generating (or is below the cost to make this feasible), you really should look at moving from self-hosted DNS.
It is simple, but it has mixed results.
There's about 10% of the internet that's using "parent centric" DNS resolvers. What that means is that when it comes to what name-servers to query for a zone; those resolvers pay more attention to what's in the parent-zone than what the child-zone lists for itself.
For that 10% of the internet; the TTL on the NS record set is out of your control - it's whatever the parent domain's policy is. Although you may have set a TTL of an hour - for nimble re-delegations on your end, you'll have to wait out the time-to-update from your registrar (usually an hour, but sometimes a business day or more, depending on the TLD) and then also update the TTL on their end. For most domains, like com and net, it's two days.
So if you need to add name-servers after a problem starts, expect it to take two+ days to fully roll out to all users.
That said, the progress follows a distribution, and you'll remediate things for many users quite sooner - but still probably not within an an availability target of even three nines per month.
There's about 10% of the internet that's using "parent centric" DNS resolvers. What that means is that when it comes to what name-servers to query for a zone; those resolvers pay more attention to what's in the parent-zone than what the child-zone lists for itself.
For that 10% of the internet; the TTL on the NS record set is out of your control - it's whatever the parent domain's policy is. Although you may have set a TTL of an hour - for nimble re-delegations on your end, you'll have to wait out the time-to-update from your registrar (usually an hour, but sometimes a business day or more, depending on the TLD) and then also update the TTL on their end. For most domains, like com and net, it's two days.
So if you need to add name-servers after a problem starts, expect it to take two+ days to fully roll out to all users.
That said, the progress follows a distribution, and you'll remediate things for many users quite sooner - but still probably not within an an availability target of even three nines per month.
This is workable for small sites, but are you really equipped to deal with a denial of service attack? There are plenty of small DNS providers (Zerigo comes to mind) that can't even stay up during attacks.
I would assume that sites affected by a Network Solutions downtime would qualify as small(ish) sites because the big ones have other solutions anyways. Also, it's much easier to deal with a DOS on DNS than with a DOS on your app servers. If you can handle it for your app servers, you'll be able to handle if for DNS.
If not, well, then you're down anyways.
If not, well, then you're down anyways.
FYI, Zerigo DNS is a joke.
They have a few boxes at SoftLayer and Linode. See my comment on their last outage: https://news.ycombinator.com/item?id=5922707
They have a few boxes at SoftLayer and Linode. See my comment on their last outage: https://news.ycombinator.com/item?id=5922707
I thought that even if you self host - isn't it your registrar that points the dns records to your dns servers? Would someone with self-hosted DNS still be working in this situation? (doesn't the DNS query first go to network solutions, who refers it to your name servers?).
I'm not trying to argue, I genuinely need to know the answer as I'll have to explain it to many other people, later today...
I'm not trying to argue, I genuinely need to know the answer as I'll have to explain it to many other people, later today...
No, they just go into the zone.
You run a server for example.net. Your server will respond to any requests sent to it for records in that zone.
Someone attempting to reach your site will query the parent zone (.com) for NS records for your zone. So they will query for NS records for example.com against a server which is authoratitive for .com.
(How do they find .com server? They ask the root servers. How do they find the root servers? That's the bootstrap info.)
You run a server for example.net. Your server will respond to any requests sent to it for records in that zone.
Someone attempting to reach your site will query the parent zone (.com) for NS records for your zone. So they will query for NS records for example.com against a server which is authoratitive for .com.
(How do they find .com server? They ask the root servers. How do they find the root servers? That's the bootstrap info.)
I self-host for most things. I live on the edge and don't have backup DNS. For most situations, you'd want backup DNS as well.
ns1.kimsal.com ns2.kimsal.com nsA.offsite1.net nsB.offsite1.net
etc. the kimsal ones are my primary/secondary, but having the others offsite, slaving from the primary, would mean that even if my primaries went down, the info would be at the 'offsite' ones. Those 'offsite' DNS servers would need to be registered with the registrar as well, and become part of your record (you can check that with a whois).
ns1.kimsal.com ns2.kimsal.com nsA.offsite1.net nsB.offsite1.net
etc. the kimsal ones are my primary/secondary, but having the others offsite, slaving from the primary, would mean that even if my primaries went down, the info would be at the 'offsite' ones. Those 'offsite' DNS servers would need to be registered with the registrar as well, and become part of your record (you can check that with a whois).
One aspect of Network Solutions business is managing some of the root zones. Those are the servers that ultimately tell the internet what other servers to go look at to resolve queries. These are the "root" nameservers. So while Network Solutions is in charge of the root zones themselves, and does run a bunch of root nameservers, there are other root nameservers run by Various organizations and collectively they can handle a massive amount of work.
That is not what is being attacked.
If somone DID successfully DDOS a large enough portion of the root nameserver infrastructure, the internet would largely grind to a halt. It is designed suitably sized, and run by large enough players that this is not likely.
The DNS service that is being attacked is the one they offer end users, usually along with domain registration. There is no particular reason you need to use their services when registering a domain with them, although, like most rigistrars, their website will make this deliberately confusing, and may require you to use their DNS service if you also want to use some of their other value-added services along with your registration (like email forwarding, websites, stuff like that).
Unless you are using other services from NetSol that require you to use their DNS, you are free to use whatever DNS infrastructure you like, including your own.
(If DDOS is a business concern, I'd go with someone big who deliberately does just DNS.. Neustar or one of those)
The DNS service that is being attacked is the one they offer end users, usually along with domain registration. There is no particular reason you need to use their services when registering a domain with them, although, like most rigistrars, their website will make this deliberately confusing, and may require you to use their DNS service if you also want to use some of their other value-added services along with your registration (like email forwarding, websites, stuff like that).
Unless you are using other services from NetSol that require you to use their DNS, you are free to use whatever DNS infrastructure you like, including your own.
(If DDOS is a business concern, I'd go with someone big who deliberately does just DNS.. Neustar or one of those)
[deleted]
No, this is for domains where .worldnic.com is the authorative server. If you registered your name through NetSol but hosted the DNS servers elsewhere then you would just be relying on the respective TLD servers (e.g. .gtld-servers.net for .com domains) to refer resolvers to the correct authorative servers.
What's important is that you treat the DNS management aspect of your business as a proper thing to be managed by itself. Whether you host it yourself, or contract it out to a service that specifically does this for you is a matter of risk tolerance and budget.
A common setup is a private nameserver that you host yourself, wherever, that isn't publicly used. You then contract UltraDNS/Neustar or their brethren and set them up as your public servers, doing zone transfers from your private ones. You get the advantages of direct management of your zones, and the global scale infrastructure needed to largely eliminate nameserver issues.
A common setup is a private nameserver that you host yourself, wherever, that isn't publicly used. You then contract UltraDNS/Neustar or their brethren and set them up as your public servers, doing zone transfers from your private ones. You get the advantages of direct management of your zones, and the global scale infrastructure needed to largely eliminate nameserver issues.
What happens when someone doesn't like you and decides to DDoS your DNS servers?
Though if it's only your site on the DNS Servers, they might as well just DDoS your website.
Though if it's only your site on the DNS Servers, they might as well just DDoS your website.
Well, what happens if someone doesn't like your registrar and DDoS' them?
Do both.
Do both.
You would hope that a registrar would have better bandwidth and/or capabilities to protect against that then what you would generally have for your server(s).
You would.. but the business of being a registrar is bout managing zone records to be served out by the root/regional nameserver infrastructure, which your registrar may not even have much of a part in.
If they run additional business in hosting DNS, websites, email, and so on, as is common - that's really got nothing to do with the setup they need to act as a registrar.
If they run additional business in hosting DNS, websites, email, and so on, as is common - that's really got nothing to do with the setup they need to act as a registrar.
Honest question from a networking beginner:
So suppose right now I've got two name servers configured, NS93.worldnic.com and NS94.worldnic.com. These are down as they're the part of the Network Solution's name servers that are having issues.
If I had added more, for instance if I used Amazon's Route53 and added two name servers of theirs in addition to the *.worldnic.com ones, would my site be reachable right now?
So suppose right now I've got two name servers configured, NS93.worldnic.com and NS94.worldnic.com. These are down as they're the part of the Network Solution's name servers that are having issues.
If I had added more, for instance if I used Amazon's Route53 and added two name servers of theirs in addition to the *.worldnic.com ones, would my site be reachable right now?
Yes, but there may be delay, because the nameservers are tried in random order, and so each server that is down will have to time out before users move on to the next.
(I work on Route 53).
We've done a lot of experiments on this one and we've found that the most common resolvers make 3 tries to 3 different servers by default. At first those servers are picked at random, but over time the resolvers usually "home in" on what the least-latent nameserver is. Once they do, and have a good round-trip-time estimate for how long it takes to respond, they stay using it. But they have a hair-trigger; if the nameserver doesn't respond, they'll very quickly fall-back to trying the other nameservers.
In practice what that means is that up to two of your nameservers may be completely unresponsive and the effects will be pretty negligible. So if you're using multiple DNS providers and want to protect against one going off-air; use no more than two nameservers from each provider.
That said, for all of the reasons above, and some more, the Route 53 SLA currently only applies if you use all four Route 53 nameservers.
We've done a lot of experiments on this one and we've found that the most common resolvers make 3 tries to 3 different servers by default. At first those servers are picked at random, but over time the resolvers usually "home in" on what the least-latent nameserver is. Once they do, and have a good round-trip-time estimate for how long it takes to respond, they stay using it. But they have a hair-trigger; if the nameserver doesn't respond, they'll very quickly fall-back to trying the other nameservers.
In practice what that means is that up to two of your nameservers may be completely unresponsive and the effects will be pretty negligible. So if you're using multiple DNS providers and want to protect against one going off-air; use no more than two nameservers from each provider.
That said, for all of the reasons above, and some more, the Route 53 SLA currently only applies if you use all four Route 53 nameservers.
Any thoughts on where/how is this 'home in' state is saved? Other the cache and filtering/acl state resolvers should be stateless, so would be interested in any details, especially if this is on a per-zone, or per-set of nameservers or etc basis. I've never seen bind9 act like this.
I haven't read the bind source in a long time, but my best guess is that it's in memory.
http://docstore.mik.ua/orelly/networking_2ndEd/dns/ch02_06.h...
Has some details, and I believe Bind and PowerDNS call it "Smoothed RTT" or SRTT.
http://docstore.mik.ua/orelly/networking_2ndEd/dns/ch02_06.h...
Has some details, and I believe Bind and PowerDNS call it "Smoothed RTT" or SRTT.
interesting, not a ton of details of how. off to the source I go, thanks. sounds like it assigns a 'score' of some sort to each authoritative ns.
This might help - section 4.4 has formal details on the Bind server selection algorithms:
http://ocw.mit.edu/courses/mathematics/18-996-topics-in-theo...
yep definitely, an interesting implementation, and certainly almost impossible to guess from passive observation. thanks.
BIND assigns a score based on every IP it talks to and a hit rate. (so if ns1.foo.com resolves to 3 IPs, each is tracked individually, same if ns1.bar.com and ns1.baz.com are the same IP) It is a pretty small cache, maybe like 1024?
This is all from fuzzy 5 year old memory.
This is all from fuzzy 5 year old memory.
It depends on the implementation of the resolver, but in general when no information is known about a nameserver, the resolver will pick one at random. Overtime, it'll keep track of the latency and pick one (usually randomly within a band specifying a range of latencies). See this paper for more info: https://www.dns-oarc.net/files/workshop-201203/OARC-workshop...
Wouldn't a delay be better than full downtime?
Also you as always have to be careful and make sure that all DNS servers have the same content or else you will get weird bugs.
In general this ought to "work", but most dns check services and uppity sysadmins will complain loudly and bitterly that there's more than one SOA if you don't do a proper primary-secondary with IXFR.
[deleted]
yes
This[0] FB post is the best I can come up with as an explanation.
> Yesterday, some Network Solutions customer sites were compromised.
The funny thing is, they have a link in their post going back to their site, which of course doesn't work.
[0]: https://www.facebook.com/networksolutions/posts/101514668014...
> Yesterday, some Network Solutions customer sites were compromised.
The funny thing is, they have a link in their post going back to their site, which of course doesn't work.
[0]: https://www.facebook.com/networksolutions/posts/101514668014...
That explains my friend's site yesterday. Requests to his site sometimes resulted in the display of a banner page by "Islamic Ghosts Team," but not all the time. I noticed that Network Solutions was apparently running Apache 2.2.22, which has a few security flaws (I'm pretty sure he doesn't use a VPS).
TechZone360 (ugh, so many ads) article with a little more information:
Network Solutions Experiences Hijacking of DNS Records http://www.techzone360.com/topics/techzone/articles/2013/07/...
Network Solutions Experiences Hijacking of DNS Records http://www.techzone360.com/topics/techzone/articles/2013/07/...
A comment in the article pointed out that this one seems very similar to a Cisco article from a month ago: http://blogs.cisco.com/security/hijacking-of-dns-records-fro...
If your name servers are *.worldnic.com it appears you are affected.
First mention on Twitter was an hour ago https://twitter.com/thefrost/status/357483942082920449
First mention on Twitter was an hour ago https://twitter.com/thefrost/status/357483942082920449
Site down? Don't link to it. Bad netiquette. Jesus christ.
Seeing this as well, appears to be DNS only at this point.
You're right. Title updated.
Our site has been sporadically up and down for the past few hours. Glad I know the culprit finally. Hope this gets resolved soon
Oh crap. Today is going to be a bad day for me.
If the mail server goes down again it'll be a bad day for us too.
We run our own mail server, but the MX records come from network solutions :(
DNS for our domains seems to be back up now, for now at least.
In case anyone is interested, looking at our google analytics it appears this started sometime around 6AM MST, and got 'fixed' around 8:45AM MST. Of course it's impossible to say exactly when, because of DNS TTL & caching.
In case anyone is interested, looking at our google analytics it appears this started sometime around 6AM MST, and got 'fixed' around 8:45AM MST. Of course it's impossible to say exactly when, because of DNS TTL & caching.
We're slowly coming back online, there are reports of a (D)DOS attack against them but I can't find a source.
Some talk here [1] of a recorded message from their support number that they are currently experiencing a DOS attack.
1. http://www.webhostingtalk.com/showthread.php?t=1285835
1. http://www.webhostingtalk.com/showthread.php?t=1285835
I've been in panic mode trying to figure out why our applications have gone down. At least now I know why.
Ditto. I just spent an hour hacking at my firewall rules before I tried the obvious thing: going to the server's IP.
It doesn't matter how long I do this... DNS problems always get me.
It doesn't matter how long I do this... DNS problems always get me.
Have you thought about setting up with an external DNS monitoring service? It's cheap and really comes in handy over the years.
And to think you paid extra for this service.
Just what I wanted to wake up to. Client emails asking me why their sites were down. Sigh.
Why did I click on the link to a down website? What did I expect?
Shame on me.
[deleted]
This appears to be a problem again this afternoon (1:44PM MST)
Their mail server appears to be online.
Back up.
Network Solutions is experiencing a Distributed Denial of Service (DDOS) attack that is impacting our customers as well as the Network Solutions site. Our technology team is working to mitigate the situation. Please check back for updates.
https://www.facebook.com/networksolutions/posts/101514681058...