The Critical Security Flaws that Resulted in Last Friday's Hack(blog.cloudflare.com)
blog.cloudflare.com
The Critical Security Flaws that Resulted in Last Friday's Hack
http://blog.cloudflare.com/the-four-critical-security-flaws-that-resulte
21 comments
I was in the same situation about 1,5 years ago, when some security hole in Gmail allowed my account to be hijacked. The account was suspended automatically after the attacker had sent out 7 spam mails, and I could reset the account again without the attacker returning ever again. But, the ping ponging did go on for a while and Google did not bother.
Good idea I think! Lock it after 2 account changes for a few hours at the very least, then the original owner will have it back, who is most likely the legitimate owner.
I would just be worried about how they lock us out, what if the block both of our IP's and then the hacker manages to change his IP and can now get in and has an advantage over the legitimate account holder.
Rule #1 when publicizing security incidents: always publish something else to the blog within 1.5 hours so that the security incident isn't the top post.
Edit: semi-tongue-in-cheek per comments below; as a CloudFlare customer I went to the blog when this first came up expecting to see something but bounced when the first post was a discussion of SSL BEAST since that was the hotness back in the fall of 2011.
I do believe it was not planned, but I also feel that vulnerability disclosures should be pinned for a while somehow if possible. I think one way this is done is having a separation between 'new feature' blog and 'ops' blog.
Edit: semi-tongue-in-cheek per comments below; as a CloudFlare customer I went to the blog when this first came up expecting to see something but bounced when the first post was a discussion of SSL BEAST since that was the hotness back in the fall of 2011.
I do believe it was not planned, but I also feel that vulnerability disclosures should be pinned for a while somehow if possible. I think one way this is done is having a separation between 'new feature' blog and 'ops' blog.
That seems kind of shady, I mean sure it sucks that it happens and it is kind of bad to have it as your top post but publishing just because you don't want it to be the latest post kinda downplays the publicity factor of being honest about what happened.
I'm sure j_s's comment was meant as tongue-in-cheek and not as an actual criticism. I have every confidence that eastdakota's response (http://news.ycombinator.com/item?id=4066982) indicating that it was just a coincidence is true.
Haha. Wish we were that organized. We have a big announcement on Wednesday and need the announcement about Polish (http://blog.cloudflare.com/introducing-polish-automatic-imag...) and the feature we're announcing tomorrow (Mirage) to come out before then.
I think being among the top stories on Hacker News will take care of people seeing it. And, for the record, I voted the breach story up.
I think being among the top stories on Hacker News will take care of people seeing it. And, for the record, I voted the breach story up.
Hey, just wondering, did someone make that infographic by hand or is there some software that does it for you? EDIT: Made by a graphic designer, answered below.
And now I will shamelessly take this moment to request a few features related to account security :-).
* Alerting: SMS or e-mail notification when an unrecognized device logs into my account or when records in my domains change.
* 2-factor Authentication: Prompt for a code delivered via SMS, e-mail, Google Authenticator, or DUO Security to login from an unrecognized device.
* Login Accounting: Let me see what IPs logged into my account, when, geoip info for each, and preferably what actions they took while logged in. Provide an API for this info so I can write an automated script to analyze it for suspicious events.
If you end up making any of these features, it would be cool to open-source a library you used to do it. There are a bunch of large SaaS providers out there that use features like these but they're all homegrown implementations afaik.
Btw, the Google Apps Admin Audit API exists but I have never seen anyone do anything with it and it makes me sad. A few hours with [name a scripting language] and you could probably have a pretty robust Google Apps monitoring system, but no one seems to care: https://developers.google.com/google-apps/admin-audit/get_st...
And now I will shamelessly take this moment to request a few features related to account security :-).
* Alerting: SMS or e-mail notification when an unrecognized device logs into my account or when records in my domains change.
* 2-factor Authentication: Prompt for a code delivered via SMS, e-mail, Google Authenticator, or DUO Security to login from an unrecognized device.
* Login Accounting: Let me see what IPs logged into my account, when, geoip info for each, and preferably what actions they took while logged in. Provide an API for this info so I can write an automated script to analyze it for suspicious events.
If you end up making any of these features, it would be cool to open-source a library you used to do it. There are a bunch of large SaaS providers out there that use features like these but they're all homegrown implementations afaik.
Btw, the Google Apps Admin Audit API exists but I have never seen anyone do anything with it and it makes me sad. A few hours with [name a scripting language] and you could probably have a pretty robust Google Apps monitoring system, but no one seems to care: https://developers.google.com/google-apps/admin-audit/get_st...
Rule #1b: always get hacked the week of WWDC so nothing stays in the news :)
Part of this attack bears a certain resemblance to the recent Bitcoinica compromise, in that, from what I understand, they were also forwarding admin emails to personal accounts, one of which was compromised leading to the attacker gaining control of the bitcoinica virtual servers. Cloudflare were fortunate - at least the changes made were reversible, whereas the bitconica compromise resulted in the virtual servers being unrecoverably deleted after breach and theft.
There are lessons to be learned from both incidents.
There are lessons to be learned from both incidents.
Does anyone else get the feeling that the attacker is going to be someone the Cloudflare team knows? Firstly they would have had to have known Matthew's phone number. Then, assuming the attacker always had the plan of disrupting the target site, they would have had to have known that the password reset mails were BCC'd to admins.
Getting someones phone number seems pretty insignificant compared to using a previously undisclosed google security flaw.
And it's probably safe to assume that once you control the admin email account for a site, it's game over. You could request resets from other providers
And it's probably safe to assume that once you control the admin email account for a site, it's game over. You could request resets from other providers
That's what I thought too. With only 2 hours in the system they seemed to know a lot of details about internal stuff.
O.o This reminds me so much of the hack sequence from the game Uplink. The game was based on "hacking" but intentionally used hacking techniques from Hollywood :) It was pretty fun. Anyway for the highest-level targets, you had to get a voiceprint from the phone of an admin, crack the password on the box, and break the encryption while bypassing monitors.
Just found out: Uplink is on Steam, and in the Ubuntu Software Center now.
Just found out: Uplink is on Steam, and in the Ubuntu Software Center now.
Incredible hack! Hats off to the hacker and to Cloudflare for the transparency of the response.
What customer was the target?
What customer was the target?
Another poster on r/netsec (https://pay.reddit.com/r/netsec/comments/ui0k4/cloudflare_wa...) mentioned that it was 4chan.org.
Wow - that's some epic transparency. Kudos to Matt and the Cloudflare team for that.
I like the infographic showing the series of events, definitely goes a long way in terms of aiding transparency. I wonder if it could be a new trend?
I would love to know if the graphic was made by hand or in an automated fashion.
Made by hand by Kevin our great graphics designer. Was up late very last night putting it together. Filled in the last few details this AM.
Its awesome, hat tip to Kevin. You guys should make the smaller version a link to the full version. I did not see the text link at first...
Posterous usually does that automatically. Not sure why it isn't here. Will see if I can wrap the image in an <a> tag.
Update: Done.
Update: Done.
Make kevin cringe less by calling him a "graphic designer". That extra "s" makes them die inside. :)
Also, nice job, kevin.
Also, nice job, kevin.
I too like it but it feels... I'm not sure how to describe it, but I felt like I was having to concentrate to read it properly and keep track of what point I was at. I'm not sure if that's a problem with me or the format, but trying to read it properly was less enjoyable than just scanning it.
Me too, especially in light of the posting on here the other day criticizing wannabe infographics. Now this is an infographic worth the name.
Would most of the attack been rendered impossible if Matthew answered his phone at 11:39 instead of letting it go to voicemail?
I'm not sure of the details here, but it wouldn't be too hard to make sure the reset call arrived in the middle of another call.
That makes a lot of sense and could have been what happened. It would also make it more difficult for Google to do something like ignore responses that come after 4+ rings.
I'm more curious why a "secure" PIN is simply left, automated, as a message. A more "secure" option, I would think, would be to require some sort of input from the person who answered (say, "Press 1 for the PIN" where that number is randomized, or something).
"AT&T was tricked into redirecting my voicemail to a fraudulent voicemail box"
Capitan Crunch called...
Capitan Crunch called...
The hack seems very well planned - I wonder just how many smaller sites have been hacked the exact same way as practice, and not picked up since they didn't have direct access to Google's security people?
I'd imagine you wouldn't blow a discovered flaw in Google's two-factor authentication setup on a small site. You'd sit on it until a big enough target came along, which CloudFlare certainly has become.
Pretty intense hack... I think Google gives you the option of setting up a recovery phone to receive voice or SMS messages. It looks like SMS may be more secure.
It also sounds like he didn't have 2 factor setup on his personal gmail account. I wonder if that would of helped.
It also sounds like he didn't have 2 factor setup on his personal gmail account. I wonder if that would of helped.
I believe that once you have it set to use a phone number, it's always possible for the attacker to choose to have it call instead of sending a text. (I seem to recall it provides that option on the 2-factor login screen.)
So, the "Five Whys" analysis came up one short, eh?
Just kidding, this is a great level of detail and much appreciated, to understand CloudFlare's process and how to protect against or recognize these tactics elsewhere.
Just kidding, this is a great level of detail and much appreciated, to understand CloudFlare's process and how to protect against or recognize these tactics elsewhere.
Is flaw # 5 (or #1 depending on how you look at it) not having two-factor auth on the personal account? Or does account recovery by-pass two-factor auth by design?
The funny part - cloudflare still hosts the DNS for the guys who claim responsibility for this attack:
Domain Name: UGNAZI.COM
Registrar: ENOM, INC.
Whois Server: whois.enom.com
Referral URL: http://www.enom.com
Name Server: LEE.NS.CLOUDFLARE.COM
Name Server: RUTH.NS.CLOUDFLARE.COM
Status: clientTransferProhibited
Updated Date: 29-may-2012
Creation Date: 22-jan-2012
Expiration Date: 22-jan-2013Great job on the transparency.
And now I know about the Google Authenticator app. Fancy little thing, that; glad to find out about it.
And now I know about the Google Authenticator app. Fancy little thing, that; glad to find out about it.
The fact that they are relying on an external mail vendor, and had passwords in their emails is a very sad practice.
I'm guessing that they didn't have passwords, but were BCC-ing the password reset emails/links to the gmail account.
But yeah... I don't like the idea of using Gmail for this either.
But yeah... I don't like the idea of using Gmail for this either.
Sounds like if the hacker had just done it out of hours, perhaps when the person in question was asleep, they would have had uncontested access to the accounts and the hack might have been far more damaging.
I'm amazed Cloudflare got a response from Google that quickly. I'm a paying Apps customer and I don't see responses for 24 to 48 hours on security incidents, not to mention that Google doesn't have a stellar reputation when it comes to things like "support".
Goes to show, it's always who you know (or it's bullshit, which is less likely). Or I don't have enough users.
Goes to show, it's always who you know (or it's bullshit, which is less likely). Or I don't have enough users.
Paying Apps customers can call Google and talk to someone immediately.
I'm surprised that Google doesn't detect two people fighting for control of one account. They could have easily detected ping-pong sessions and and locked both parties out of their accounts for a couple of hours. Or they could have penalized the newly added recovery address by forcing an exponential delay between resets using that address. This is not the first time I've heard of somebody breaking into a Gmail account while the account owner is using that very same account.