DDoS Attack Against Dyn Managed DNS(dynstatus.com)
dynstatus.com
DDoS Attack Against Dyn Managed DNS
https://www.dynstatus.com/incidents/nlr4yrr162t8
674 comments
Relevant (or at least a-propos) post by Bruce Schneier, from a month ago: "Someone Is Learning How to Take Down the Internet"
https://www.schneier.com/blog/archives/2016/09/someone_is_le...
Edit: And to be clear: I don't mean to imply there's any connection :)
https://www.schneier.com/blog/archives/2016/09/someone_is_le...
Edit: And to be clear: I don't mean to imply there's any connection :)
I wanted to provide an update on the PagerDuty service. At this time we have been able to restore the service by migrating to our secondary DNS provider. If you are still experiencing issues reaching any pagerduty.com addresses, please flush your DNS cache. This should restore your access to the service. We are actively monitoring our service and are working to resolve any outstanding issues. We sincerely apologize for the inconvenience and thank our customers for their support and patience. Real-time updates on all incidents can be found on our status page and on Twitter at @pagerdutyops and @pagerduty. In case of outages with our regular communications channels, we will update you via email directly.
In addition you can reach out to our customer support team at [email protected] or +1 (844) 700-3889.
Tim Armandpour, SVP of Product Development, PagerDuty
In addition you can reach out to our customer support team at [email protected] or +1 (844) 700-3889.
Tim Armandpour, SVP of Product Development, PagerDuty
I'm a GitHub employee and want to let everyone know we're aware of the problems this incident is causing and are actively working to mitigate the impact.
"A global event is affecting an upstream DNS provider. GitHub services may be intermittently available at this time." is the content from our latest status update on Twitter (https://twitter.com/githubstatus/status/789452827269664769). Reposted here since some people are having problems resolving Twitter domains as well.
"A global event is affecting an upstream DNS provider. GitHub services may be intermittently available at this time." is the content from our latest status update on Twitter (https://twitter.com/githubstatus/status/789452827269664769). Reposted here since some people are having problems resolving Twitter domains as well.
To get on github you can add to your /etc/hosts:
Edit; for profile pics include:
192.30.253.113 github.com
151.101.32.133 assets-cdn.github.com
Edit; for profile pics include:
151.101.32.133 avatars0.githubusercontent.com
151.101.32.133 avatars1.githubusercontent.com
151.101.32.133 avatars2.githubusercontent.com
151.101.32.133 avatars3.githubusercontent.com
151.101.32.133 avatars4.githubusercontent.com
151.101.32.133 avatars5.githubusercontent.comSo who was prepared for this? Pornhub:
pornhub.com:
pornhub.com:
Name Server: ns1.p44.dynect.net
Name Server: ns2.p44.dynect.net
Name Server: ns3.p44.dynect.net
Name Server: ns4.p44.dynect.net
Name Server: sdns3.ultradns.biz
Name Server: sdns3.ultradns.com
Name Server: sdns3.ultradns.net
Name Server: sdns3.ultradns.org
Name Server: PDNS196.ULTRADNS.ORG
Name Server: ARI.ALPHA.ARIDNS.NET.AU
Name Server: ARI.BETA.ARIDNS.NET.AU
Name Server: ARI.GAMMA.ARIDNS.NET.AU
Name Server: ARI.DELTA.ARIDNS.NET.AU
Name Server: PDNS196.ULTRADNS.NET
Name Server: PDNS196.ULTRADNS.COM
Name Server: PDNS196.ULTRADNS.BIZ
Name Server: PDNS196.ULTRADNS.INFO
Name Server: PDNS196.ULTRADNS.CO.UKI was not aware of the attacks going on until this happened:
1. Tried to download "Unknown Horizons" (game featured recently on Hacker News) binary, github-link doesn't work.
2. Think "Ok, might be an old link", google their github-repository, github appears down.
3. Try accessing github status website, is down.
4. Interested, try to visit github status twitter account, twitter is down.
Really weird experience, normally at least the second source of news on a downed website I try during an attack works.
1. Tried to download "Unknown Horizons" (game featured recently on Hacker News) binary, github-link doesn't work.
2. Think "Ok, might be an old link", google their github-repository, github appears down.
3. Try accessing github status website, is down.
4. Interested, try to visit github status twitter account, twitter is down.
Really weird experience, normally at least the second source of news on a downed website I try during an attack works.
According to Fortune, Hacker News "reported" on the incident. Are we journalists now?
"Popular tech site Hacker News reported many other sites were affected including Etsy, Spotify, Github, Soundcloud, and Heroku." -- http://fortune.com/2016/10/21/internet-outages/
"Popular tech site Hacker News reported many other sites were affected including Etsy, Spotify, Github, Soundcloud, and Heroku." -- http://fortune.com/2016/10/21/internet-outages/
Very funny guys, can you stop now? We have a demo in 4 minutes.
I can't currently get resolution on www.paypal.com.
$ dig @8.8.8.8 www.paypal.com
; <<>> DiG 9.8.1-P1 <<>> @8.8.8.8 www.paypal.com ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 17925 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION: ;www.paypal.com. IN A
;; Query time: 29 msec ;; SERVER: 8.8.8.8#53(8.8.8.8) ;; WHEN: Fri Oct 21 12:35:33 2016 ;; MSG SIZE rcvd: 32
$ dig @8.8.8.8 www.paypal.com
; <<>> DiG 9.8.1-P1 <<>> @8.8.8.8 www.paypal.com ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 17925 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION: ;www.paypal.com. IN A
;; Query time: 29 msec ;; SERVER: 8.8.8.8#53(8.8.8.8) ;; WHEN: Fri Oct 21 12:35:33 2016 ;; MSG SIZE rcvd: 32
I am confused. Are so many big websites using Dyn, or does Dyn have some special role in the DNS chain in the US?
I'm updating a list of confirmed outages as I see them here https://news.ycombinator.com/item?id=12759520
So far twitter, etsy, soundcloud, spotify, github, pagerduty...crazy that this can even happen
So far twitter, etsy, soundcloud, spotify, github, pagerduty...crazy that this can even happen
Journalist and security researcher Brian Krebs believes this is someone doing a DDoS as payback for research into questionable "DDoS mitigation services" that he and Dyn's Doug Madory did. Doug just presented his results yesterday at NANOG and Krebs believes this is payback. Read more: https://krebsonsecurity.com/2016/10/ddos-on-dyn-impacts-twit...
I'm wondering, from a regulatory perspective, what might be done to mitigate DDoS attacks in the future?
From comments made on this and other similar posts in the past, I've gathered the following:
1) Malicious traffic often uses a spoofed IP address, which is detectable by ISPs. What if ISPs were not allowed to forward such traffic?
2) There is no way for a service to exert back pressure. What if there was? e.g. send a response indicating the request was malicious (or simply unwanted due to current traffic levels), and a router along the way would refuse to send follow up requests for some time. There is HTTP status code 429, but that is entirely dependent on a well-behaved client. I'm talking about something at the packet level, enforced by every hop along the way.
3) I believe it is suspected that a substantial portion of the traffic is from compromised IoT devices. What if IoT devices were required to continually pass some sort of a health check to make other HTTP requests? This could be enforced at the hardware/firmware level (much harder to change with malware), and, say, send a signature of the currently running binary (or binaries) to a remote server which gave the thumbs up/down.
From comments made on this and other similar posts in the past, I've gathered the following:
1) Malicious traffic often uses a spoofed IP address, which is detectable by ISPs. What if ISPs were not allowed to forward such traffic?
2) There is no way for a service to exert back pressure. What if there was? e.g. send a response indicating the request was malicious (or simply unwanted due to current traffic levels), and a router along the way would refuse to send follow up requests for some time. There is HTTP status code 429, but that is entirely dependent on a well-behaved client. I'm talking about something at the packet level, enforced by every hop along the way.
3) I believe it is suspected that a substantial portion of the traffic is from compromised IoT devices. What if IoT devices were required to continually pass some sort of a health check to make other HTTP requests? This could be enforced at the hardware/firmware level (much harder to change with malware), and, say, send a signature of the currently running binary (or binaries) to a remote server which gave the thumbs up/down.
Analysis of the Mirai botnet: [1]
This is worth reading. It has links to copies of the code and names the known control servers. Quite a bit is known now about how this thing works.
The bots talk to control servers and report servers. The attacker appears to communicate with the report servers over Tor.
[1] http://blog.level3.com/security/grinch-stole-iot/
This is worth reading. It has links to copies of the code and names the known control servers. Quite a bit is known now about how this thing works.
The bots talk to control servers and report servers. The attacker appears to communicate with the report servers over Tor.
[1] http://blog.level3.com/security/grinch-stole-iot/
Although I don't like to to recommend Google products, they provide a provide a public DNS-over-HTTPS interface that should be useful for people who want to add specific entries into their /etc/hosts files: https://dns.google.com/query?name=github.com&type=A&dnssec=t...
"digikey.com", the big electronic part distributor, is currently inaccessible. DNS lookups are failing with SERVFAIL. Even the Google DNS server (8.8.8.8) can't resolve that domain. Their DNS servers are "ns1.p10.dynect.net" through "ns4.p10.dynect.net", so it's a Dyn problem.
This will cause supply-chain disruption for manufacturers using DigiKey for just-in-time supply.
(justdownforme.com says the site is down, but downforeveryoneorjustme.com says it's up. They're probably caching DNS locally.)
This will cause supply-chain disruption for manufacturers using DigiKey for just-in-time supply.
(justdownforme.com says the site is down, but downforeveryoneorjustme.com says it's up. They're probably caching DNS locally.)
Switch to OpenDNS servers - 208.67.222.222 and 208.67.220.220. Even google NS are down it seems. Heroku works after switching to opendns.
If you're having issues with people accessing your running Heroku apps, it's likely because you're running your DNS through herokussl.com (with their SSL endpoint product) which is hosted on Dyn.
If you can update your DNS to CNAME directly to the ELB behind it, it should at least make your site accessible.
If you can update your DNS to CNAME directly to the ELB behind it, it should at least make your site accessible.
Just to be clear, this is a DDoS against Dynect's NS hosts, right?
I'm confused because of the use of "dyn dns", which to me means dns for hosts that don't have static ip addresses.
I'm actually surprised so many big-name sites rely on Dynect, which I hadn't heard of, but more importantly don't seem to use someone else's NS hosts as 2nd or 4th entries.
I'm confused because of the use of "dyn dns", which to me means dns for hosts that don't have static ip addresses.
I'm actually surprised so many big-name sites rely on Dynect, which I hadn't heard of, but more importantly don't seem to use someone else's NS hosts as 2nd or 4th entries.
Twitter and Github are still down here in LA (and confirmed on isup.me)
OpenDNS servers seem the only ones that still work. Kudos.
It may not be the proper action but this kind of soft-fail scenario (use the old DNS until you can contact the DNS servers and get new ones) is much better.
It may not be the proper action but this kind of soft-fail scenario (use the old DNS until you can contact the DNS servers and get new ones) is much better.
echo "nameserver 208.67.222.222" | sudo tee -a /etc/resolv.confAWS says "We are investigating elevated errors resolving the DNS hostnames used to access some AWS services in the US-EAST-1 Region." Is that coincidental, or are they being DDoSed also?
Anyone else spend the morning thinking the problem was their setup? I've been flushing my system DNS cache, Chrome's DNS cache, changing DNS servers, rebooting my router, turning VPN on/off, etc.
I've been singing the praise of AWS Route53 for a long time, they up and running. I can't believe major multi-million dollar companies (Twitter, GitHub, Soundcloud, Pagerduty) would not run a mix of multiple DNS providers.
Also what is happening is a cascade effect, where a 3rd party being down effects others.
Also what is happening is a cascade effect, where a 3rd party being down effects others.
OpenDNS DNS Servers (208.67.222.222 and 208.67.220.220) are still resolving websites while my typical fallback to 8.8.8.8 is not.
Twitter, Reddit, wow. I was so confused for a moment. Thankfully HN is here to explain.
Seems to be impacting POPs in US East most severly. We use Ripe Atlas to assess the impact of DNS outages, and in the past hour have measured about 50-60% recursive query failure from a few hundred probes in that region: https://cloudharmony.com/status-for-dyn
Is it time for everyone to actually start using secondary name servers/DNS resolvers too from a different provider from primary? DNS _is_ built for this, for the very purpose of handling failure of the primary resolver, isn't it? Just most people don't seem to do it -- including major players?
Or would that not actually solve this particular scenario?
Or would that not actually solve this particular scenario?
Heroku also seems to be affected. I'm getting this when I run 'heroku status':
>> We are seeing a widespread DNS issue affecting connections to our services both internally and externally.
>> We are seeing a widespread DNS issue affecting connections to our services both internally and externally.
Ideally, then, the local resolvers of the nodes and/or the UIs of applications could detect the last-known-good flag on resolution and present a UI to users ("DNS authority for this domain is unresponsive; you are visiting a last-known-good IP provided by a resolution from 8 hours ago."). But that would be a nicety, and not strictly necessary.
Is there a spectacular downside to doing so? Since the last-known-good resolution would only be used if a TTL-specified refresh failed, I don't see much downside.