Details Behind Today's Internet Hacks(blog.cloudflare.com)
blog.cloudflare.com
Details Behind Today's Internet Hacks
http://blog.cloudflare.com/details-behind-todays-internet-hacks
13 comments
I'd suggest extending the idea to non-responsive sites as well. Instead of submitting a link to a company's homepage when they're being DDOSed or are otherwise unavailable, submit a link to their status page if they have one or, failing that, use a third-party indicator. Off the top of my head I can't think of a third-party indicator that would capture point-in-time availability, but a manually crafted URL like http://isup.me/example.com?1970-01-01T00:00:00Z would get the point across just fine.
Maybe something like Zapier's API Status Board, though I'm not sure how realtime and whether it only applies to site's APIs rather than their general websites.
https://zapier.com/status/
https://zapier.com/status/
It applies to their APIs specifically. It is realtime within ~5 minutes.
Basically zero information. They keep telling us how MelbourneIT is usually more secure but doesn't do on to tell us how it is any more secure than other registrars. More importantly, even with admin access to to their control panel how can it be so easy to change registry information of such high profile sites with a click of a button?
Devilishly clever marketing for Cloudflare, though. Clearly I need to spend my days on more bridge calls for situations affecting other ops teams that have nothing to do with me, so my company can put out a PR piece from a position of authority about how awesome we are. What exactly did a team of people at Cloudflare do today? Consult? Do you bill hourly or is it a friendly NYT discount? What was your plan connecting end users with recursive operators? Want them to manually flush their resolvers out of the normal DNS TTL protocol? Is that a service that comes with my Cloudflare subscription?
Next time a startup goes down, ask yourself: if I were on a bridge call with their ops team, could I use this to sell my company's reliability product? Clearly, the answer is yes.
Classy, too, jumping out in front of MelbourneIT's response then speculating on it. I would be furious about Cloudflare writing a details-thin "postmortem," headlining it as a postmortem, analyzing my initial statement to customers in it, then getting it on HN before DNS caches are even cold from the incident itself. It's not even subtle.
This is the sort of thing I remember in discussions about using Cloudflare. There's lots of choices for CDNs, a market growing surprisingly full of ambulance chasers: one CDN startup had the fucking courage to email me directly after a hellish multi-hour outage and say "want to set up a call to discuss how our product could have prevented this outage?" I was still awake from fixing the problem overnight and no, your CDN is not going to fix my catastrophic DB failure. Get bent.
This is a disgusting move by Cloudflare. The little human network signoff made me gag; don't forget, small ops teams, you will only get things done if you know people. Notice HuffPo wasn't on the call? Exactly.
Next time a startup goes down, ask yourself: if I were on a bridge call with their ops team, could I use this to sell my company's reliability product? Clearly, the answer is yes.
Classy, too, jumping out in front of MelbourneIT's response then speculating on it. I would be furious about Cloudflare writing a details-thin "postmortem," headlining it as a postmortem, analyzing my initial statement to customers in it, then getting it on HN before DNS caches are even cold from the incident itself. It's not even subtle.
This is the sort of thing I remember in discussions about using Cloudflare. There's lots of choices for CDNs, a market growing surprisingly full of ambulance chasers: one CDN startup had the fucking courage to email me directly after a hellish multi-hour outage and say "want to set up a call to discuss how our product could have prevented this outage?" I was still awake from fixing the problem overnight and no, your CDN is not going to fix my catastrophic DB failure. Get bent.
This is a disgusting move by Cloudflare. The little human network signoff made me gag; don't forget, small ops teams, you will only get things done if you know people. Notice HuffPo wasn't on the call? Exactly.
No shit... it almost reads like a hit-piece on MelbourneIT (damning with faint praise).
But at least they rode in with some knights from the mighty Google and OpenDNS to patch some caching issues and release a State of the Domains address.
Meanwhile the empires of NYT and Twitter were left being ravaged by hordes of Syrian Ninjas and an overseas registrar.
But at least they rode in with some knights from the mighty Google and OpenDNS to patch some caching issues and release a State of the Domains address.
Meanwhile the empires of NYT and Twitter were left being ravaged by hordes of Syrian Ninjas and an overseas registrar.
Didn't they pull a similar story telling people an attack on them by Cyberbunker impacted the London Internet Exchange, prompting quite some pandemonium?
I remember there being a more somber post after the whole incident by another blog detailing just how little fluctations there were on the alleged day of the incident, and how the numbers didn't stack up.
Cloudflare is tricky, isn't it?
I remember there being a more somber post after the whole incident by another blog detailing just how little fluctations there were on the alleged day of the incident, and how the numbers didn't stack up.
Cloudflare is tricky, isn't it?
This is known as 'inserting yourself in the news story' and it works well as a marketing trick but in this case cloudflare is actually part of the story because the NYT (one of the affected sites) and cloudflare did communicate on the subject. The more peripheral the link the trickier it is, in this case (a first order contact with the affected party which was initiated by cloudflare) I think it is fine to issue some statement, but not necessarily this statement.
Not only did CloudFlare (where I work) and the New York Times communicate, the CTO of the paper has said the following: https://twitter.com/rajivpant/status/372559771960098816
"I'm super impressed by the operations, incident/crisis management & expertise of the @CloudFlare and @OpenDNS teams."
"I'm super impressed by the operations, incident/crisis management & expertise of the @CloudFlare and @OpenDNS teams."
Since it isn't obvious from your profile or the comment maybe add a bit that you're working for cloudflare?
edit: thanks John.
edit: thanks John.
Before leaving my comment, I searched and searched for any shred of reason for CloudFlare to release this inappropriate statement, including reading all of Rajiv's timeline. Obviously, since I left the comment, I came up empty.
Can you point to what you feel makes this statement appropriate on behalf of your company? I can't identify what annoys me most about it, because there are many things: the "it's who you know in ops" attitude that I've been fighting for my entire career, the creation of a Batman-esque hero at a startup CDN provider who assembles a team to guide the lesser ops teams through a crisis, the overdramatizing of a DNS hijack that happens countless times daily (just with an interesting vector this time, but certainly not the first of ITS kind, either), speculating on another company's statements, preempting an official response with your own "postmortem" to score some traffic...
It's particularly frustrating because I've been in this exact scenario, to the T and including a registrar compromise, before. But because my personal side project doesn't have name pull, I didn't get a CloudFlare Crack Squad on speakerphone calling in a dozen courtesy phone favors to score my contract. And I had to wait for tickets and TTLs like everyone else. That sounds bitter -- and I hate bringing it up for that reason -- but that's why this is ethically shitty. Either you're playing favorites or capitalizing on something for sales. There is no third option, not even an altruistic one.
Can you point to what you feel makes this statement appropriate on behalf of your company? I can't identify what annoys me most about it, because there are many things: the "it's who you know in ops" attitude that I've been fighting for my entire career, the creation of a Batman-esque hero at a startup CDN provider who assembles a team to guide the lesser ops teams through a crisis, the overdramatizing of a DNS hijack that happens countless times daily (just with an interesting vector this time, but certainly not the first of ITS kind, either), speculating on another company's statements, preempting an official response with your own "postmortem" to score some traffic...
It's particularly frustrating because I've been in this exact scenario, to the T and including a registrar compromise, before. But because my personal side project doesn't have name pull, I didn't get a CloudFlare Crack Squad on speakerphone calling in a dozen courtesy phone favors to score my contract. And I had to wait for tickets and TTLs like everyone else. That sounds bitter -- and I hate bringing it up for that reason -- but that's why this is ethically shitty. Either you're playing favorites or capitalizing on something for sales. There is no third option, not even an altruistic one.
No good deed, it seems, goes unpunished by those upset they're not getting enough attention. May I suggest you read the end of the NYT CTO's recently updated blog post:
http://www.rajiv.com/blog/2009/12/10/tech-ops-irc/#2013Aug28
http://www.rajiv.com/blog/2009/12/10/tech-ops-irc/#2013Aug28
That wasn't remotely the thrust of my comments and you know it. I also (correctly) predicted you would hop on the bitter swan song instead of, you know, the half-dozen reasons why this sucks immediately prefacing it. Also, that's two employees who have posted Rajiv's words as rationale for the blog post; can we go for three? Shouldn't you be hiring Rajiv at this point, as hard as you're riding him?
Address something smaller and bite-sized, like preempting MelbourneIT's statement with your own and speculating on their behalf. Can you at least defend that inappropriateness? Can we start there?
Your company provided guidance and connections, which makes this statement inappropriate. Or did CloudFlare do something that has been left out of all statements?
I am not annoyed by your "good deed". I'm annoyed by how hard and how inappropriately you are capitalizing upon it as a PR coup, before the ashes have even settled. The victim tone is discouraging for this conversation, I have to say, and it's quite unbecoming.
Address something smaller and bite-sized, like preempting MelbourneIT's statement with your own and speculating on their behalf. Can you at least defend that inappropriateness? Can we start there?
Your company provided guidance and connections, which makes this statement inappropriate. Or did CloudFlare do something that has been left out of all statements?
I am not annoyed by your "good deed". I'm annoyed by how hard and how inappropriately you are capitalizing upon it as a PR coup, before the ashes have even settled. The victim tone is discouraging for this conversation, I have to say, and it's quite unbecoming.
> At 1:19pm (PDT) today, a researcher noticed that the New York Times' website wasn't loading.
So if the content on the redirected page had been more subtle - for example, mirroring NYTimes but editing stories etc - then things would have taken a lot longer to have been noticed?
So if the content on the redirected page had been more subtle - for example, mirroring NYTimes but editing stories etc - then things would have taken a lot longer to have been noticed?
Are there any registrars that allow one to set serverDeleteProhibited, serverTransferProhibited, and serverUpdateProhibited?
Probably $$$$, but ....
http://reports.internic.net/cgi/whois?whois_nic=microsoft.co...
http://reports.internic.net/cgi/whois?whois_nic=microsoft.co...
Domain Name: MICROSOFT.COM
Registrar: MARKMONITOR INC.
Whois Server: whois.markmonitor.com
Referral URL: http://www.markmonitor.com
Name Server: NS1.MSFT.NET
Name Server: NS2.MSFT.NET
Name Server: NS3.MSFT.NET
Name Server: NS4.MSFT.NET
Name Server: NS5.MSFT.NET
Status: clientDeleteProhibited
Status: clientTransferProhibited
Status: clientUpdateProhibited
Status: serverDeleteProhibited
Status: serverTransferProhibited
Status: serverUpdateProhibited
Updated Date: 09-aug-2011
Creation Date: 02-may-1991
Expiration Date: 03-may-2021I'm not sure it would have helped in this instance. If the attacker got access to the administrative interface for the registrar, all he'd have to do is unset the relevant flag first, using the same interface, before changing the name server records.
These flags are the functional equivalent of forcing you to break a piece of glass before pushing the fire alarm button.
These flags are the functional equivalent of forcing you to break a piece of glass before pushing the fire alarm button.
Some registrars require you to send in a hardcopy request, along various forms of ID, in order to clear these flags. I know mine (PairNIC) does.
It's not unspoofable, but it is an time consuming extra step that involves a human on the receiving end.
Edit: I was thinking of the client(Update|Transfer|Delete)Prohibited flags, which is a registrar lock. I'm not even sure how one goes about setting the "server" version of those flags for a registry lock, but it's probably even more complicated.
It's not unspoofable, but it is an time consuming extra step that involves a human on the receiving end.
Edit: I was thinking of the client(Update|Transfer|Delete)Prohibited flags, which is a registrar lock. I'm not even sure how one goes about setting the "server" version of those flags for a registry lock, but it's probably even more complicated.
Yes. CloudFlare has it set that way with Network Solutions.
I believe MelbourneIT do (twitter.com has it set and they are registered there)
So, if my DNS is hacked, I can call Google and OpenDNS and have them correct my records upstream? And then contact Verisign for a registry lock? And expect a personal response from MelbourneIT (even though it's likely their reseller's fault)? This is great news!
If you're the paper of record, yes.
If your personal DNS is hacked? Probably not.
But if your entire registry is hacked? Probably yes, assuming you have sufficient credibility for them to notice you.
But if your entire registry is hacked? Probably yes, assuming you have sufficient credibility for them to notice you.
The details actually look pretty sparse. I'm looking forward to MelbourneIT letting us know the specifics (if they do!).
Particularly the malware. Is this related to the Google Palestine hacking yesterday? Somebody linked the hacked site and it hit top of HN, so a number of us had to have clicked through to it.
The malware makes my wonder if the "Syrian" stuff is just a cover?
It's always about the money. I learned that from Die Hard 3.
It's always about the money. I learned that from Die Hard 3.
I'm amazed that Melbourne IT seem to be held in high regard these days. Going back to the 1990s, they had a monopoly on Australian domain registration, they charged the earth, and had really crap customer service.
I've only heard bad things about them.
> The correct name servers should have been DNS.EWR1.NYTIMES.COM and DNS.SEA1.NYTIMES.COM.
How does this work? How would you get to DNS.EWR1.NYTIMES.COM without first knowing where nytimes.com is?
How does this work? How would you get to DNS.EWR1.NYTIMES.COM without first knowing where nytimes.com is?
Nameservers have their IPs registered with the registry, and they are returned in the additional answers section. These are called "glue records".
The top level servers (for .com in this case) has A (and/or AAAA) records for the name servers to prevent this kind of catch-22. These out-of zone records are called glue records.
How would setting the registrar lock have helped in this case? The registrar lock can be unlocked by the current registrar... which was the target in this case.
It's good advice, but seems kind of irrelevant.
> It's worth noting that while some of Twitter's utility domains were redirected, Twitter.com was not -- and Twitter.com has a registry lock in place.
It's good advice, but seems kind of irrelevant.
> It's worth noting that while some of Twitter's utility domains were redirected, Twitter.com was not -- and Twitter.com has a registry lock in place.
registry lock != registrar lock
The former is with Verisign and cannot easily be removed by the registrar. The latter is with the registrar and can be removed by the registrar. In whois status codes "clientXXX" = registrar lock (weak). "serverXXX" = registry lock (stronger).
The former is with Verisign and cannot easily be removed by the registrar. The latter is with the registrar and can be removed by the registrar. In whois status codes "clientXXX" = registrar lock (weak). "serverXXX" = registry lock (stronger).
I'll bet five dollars the credentials were stolen by a botnet the SEA runs or has access to. You wouldn't believe the shit that pops up sometimes. (It's also incredibly trivial to take over botnets run by jackasses who took a tutorial in setting up Zeus) Less likely but still highly possible would be spear phishing of registrar resellers.
Edit: I don't know why, but the nameservers I use don't resolve any address for nytimes.com now. If I query 8.8.8.8 directly I get a response. So, could be they're still suffering from this attack, which sucks.
Edit: I don't know why, but the nameservers I use don't resolve any address for nytimes.com now. If I query 8.8.8.8 directly I get a response. So, could be they're still suffering from this attack, which sucks.
> MelbourneIT has traditionally been known as one of the more secure registrars
They were one of the registrars compromised back in May as part of Hack the Planet[1]. If I recall correctly, they were the only registrar where the attackers actually got shell access on a server. That's when they lost any reputation for security in my eyes.
[1] http://www.theregister.co.uk/2013/05/09/melbourne_it_hacking...
They were one of the registrars compromised back in May as part of Hack the Planet[1]. If I recall correctly, they were the only registrar where the attackers actually got shell access on a server. That's when they lost any reputation for security in my eyes.
[1] http://www.theregister.co.uk/2013/05/09/melbourne_it_hacking...
I don't think I understand why CloudFlare was involved - do they provide services to NYT, it isn't clear from the post that they do.
No, they're the concerned citizen that performed first aid on the motorist, then hung around to take questions from the media.
Good to see the MelbsIT product using two factor auth.
On the HN post "Google.ps domain was hacked (google.ps)" [1], HN user biot predicted this exact scenario, although not a zero day most likely. He talked about submitting hacked sites to HN "... and thousands of HN readers get infected by a zero-day exploit. Maybe. If you're thinking of submitting a known compromised site to HN, consider instead submitting a third-party site which explains/documents the compromise. Ideally from a respected security research company". [2]
[1] https://news.ycombinator.com/item?id=6278737
[2] https://news.ycombinator.com/item?id=6279253