Large DDoS hitting CloudFlare
13 comments
CloudFlare, a product supposed to increase uptime, has been my product's only source of downtime for the past several months.
CloudFlare also caused huge performance issues on a side project which I didn't notice until recently. I was consistently getting 1.2s+ responses on a couple of pages, and 600ms on a completely static page.
So, not only did CloudFlare not help on the pages that were truly static, it was actually making everything worse across the board. Hitting the same site with my direct.* DNS cut from 1.2s+ to under 500ms.
CopperEgg and others also kept reporting that my site was down or otherwise super slow when on CloudFlare. I'm not sure if this is because of throttling or something else that might have been in place with the CF service, but either way I was often unable to reproduce the "down"/"more than 6s" that CopperEgg kept reporting.
In short, I think CloudFlare's services sound good in theory, but I'm not sure they have figure out all the issues with scaling and performance yet. So, I cut them out... and certainly not going to be paying any time soon.
So, not only did CloudFlare not help on the pages that were truly static, it was actually making everything worse across the board. Hitting the same site with my direct.* DNS cut from 1.2s+ to under 500ms.
CopperEgg and others also kept reporting that my site was down or otherwise super slow when on CloudFlare. I'm not sure if this is because of throttling or something else that might have been in place with the CF service, but either way I was often unable to reproduce the "down"/"more than 6s" that CopperEgg kept reporting.
In short, I think CloudFlare's services sound good in theory, but I'm not sure they have figure out all the issues with scaling and performance yet. So, I cut them out... and certainly not going to be paying any time soon.
I've been on the free account for a while now, and when I received a surge of traffic they responded to 460,000 of the 500,000-ish HTTP requests and served 48gb of the 50 that were requested over the course of about 12 hours. My server load sat at 0.01 the whole time and I could not have been happier. The concept is absolutely sound in theory and in practice.
You raise a great point about reliability however. The above example was a personal project so I don't really lose much if there's the odd down-time here and there. I'd love to recommend this for the enterprise-level clients where I work but the idea of a point of failure out of one's direct control is a bit alarming, and I am surprised and disappointed to see them succumb to a DDOS that affects what (I assume?) was their entire network - even paying customers.
I notice even now my homepage is taking seconds to load, when usually it responds instantly. My guess is that they're prioritizing access at this stage due to the DDOS, if it's still ongoing.
You raise a great point about reliability however. The above example was a personal project so I don't really lose much if there's the odd down-time here and there. I'd love to recommend this for the enterprise-level clients where I work but the idea of a point of failure out of one's direct control is a bit alarming, and I am surprised and disappointed to see them succumb to a DDOS that affects what (I assume?) was their entire network - even paying customers.
I notice even now my homepage is taking seconds to load, when usually it responds instantly. My guess is that they're prioritizing access at this stage due to the DDOS, if it's still ongoing.
Details?
This is why most CDNs avoid taking on controversial customers like 4chan, kaddafi.hk and other sites trafficking in illegal content. Due to their decision to accept anyone, Cloudflare is a constant target for DDOS attacks and their legitimate users occasionally share the brunt of those attacks.
4chan "trafficking in illegal content"? I don't think that's a fair assessment. They specifically censor much less than most website operators but they aren't "trafficking" any more than highway underpasses "traffic" in illegal graffiti.
I was referring more to the numerous sites they host like kaddafi.hk which sells stolen credit cards online. 4chan has a recurring problem with child porn (which their moderators do their best to prevent), but I agree with your distinction regarding trafficking.
What bothers me is that our CDN gets a lot of downtime whenever CloudFire gets hit. They're totally separate services, or at least they should be, but I always know to watch out when I see articles like these.
Which CDN do you use?
The attack directed at CloudFlare has been mitigated at this time. Performance should be back at normal levels now. https://twitter.com/CloudFlareSys/status/433017584670093312
I've had to move my blog away from EC2 because Cloudflare was killing it (it was a very simple wordpress install). Other more custom sites on other servers are doing fine though.
They manage so much stuff that any downtime they have will make a lot of people unhappy. It already happened once (http://oneurl.me/cloudflare-broke-the-internet)
They manage so much stuff that any downtime they have will make a lot of people unhappy. It already happened once (http://oneurl.me/cloudflare-broke-the-internet)
And here I was wondering why my internet (Germany) was very unstable an hour ago.
Looks like it's related: https://twitter.com/CloudFlareSys/status/432997463562022912
Looks like it's related: https://twitter.com/CloudFlareSys/status/432997463562022912
Here at 1355 Market, the Twitter building, AboveNet is entirely down. They are claiming it's a main router issue. No ETOR.
networksolutions.com is down... As well as a large merchandising site I'm working with that's DNS goes through network solutions...
Is there an economic incentive for DDoS attacks? Or is it just some bored script kiddie with nothing better to do?
My theory is that some of them are proof-of-concept attacks: "We took down X for forty eight hours; give us $Y and we'll do the same to anyone you name."
We're seeing an increase in 'down' notifications in general here at NodePing.
I would kill for a Null Route REST API at peering points/network upstream providers.
The reason the internet is still online is because people who would ask for this type of thing don't have access to it.
Routers shouldn't speak HTTP. People who don't know how to use blackhole communities have no business controlling them.
Routers shouldn't speak HTTP. People who don't know how to use blackhole communities have no business controlling them.
People who don't know how to use blackhole communities have no business controlling them.
What's a "blackhole community"? (I did Google, but found email messages from 2003 on page one. Surely there are better references?)
What's a "blackhole community"? (I did Google, but found email messages from 2003 on page one. Surely there are better references?)
Thank you, commenters like you are why the HN community rocks.
You're welcome! Not a problem at all.
Your routers wouldn't speak HTTP. Your web service endpoint would talk to your network management middleware, which would then issue your respective IOS/NXOS/JunOS commands to your core or edge gear.
You'd grant your customers the ability to null route traffic from IP blocks (/24 or larger, because ain't nobody got memory to route blocks smaller than that in IPv4) so they wouldn't saturate their links with useless traffic. There was a discussion on the North American Network Operators Group (NANOG) mailing list a few weeks ago.
Disclaimer: I have operated large-scale networks for over a decade.
You'd grant your customers the ability to null route traffic from IP blocks (/24 or larger, because ain't nobody got memory to route blocks smaller than that in IPv4) so they wouldn't saturate their links with useless traffic. There was a discussion on the North American Network Operators Group (NANOG) mailing list a few weeks ago.
Disclaimer: I have operated large-scale networks for over a decade.
Sorry, but I call shenanigans on your having operated a large scale network.
Every carrier worth its salt will already let you use blackhole communities to mitigate attacks. You tag it, it gets dropped at the edge of your upstreams networks. Simple and effective. You don't need a web service or middleware for any of this.
Also, a route and netmask (generally) take exactly the same amount of memory regardless of the size of the network you're covering.
Every carrier worth its salt will already let you use blackhole communities to mitigate attacks. You tag it, it gets dropped at the edge of your upstreams networks. Simple and effective. You don't need a web service or middleware for any of this.
Also, a route and netmask (generally) take exactly the same amount of memory regardless of the size of the network you're covering.
If you want to meet me at the next NANOG conference, I can send you my personal email address to get in touch.
[deleted]
We would people try to do a ddos on them. A ddos is a temporary thing. Only time it is worth it do do a ddos is when there is some kind of event thing.
Like if some one wants to get a site down today so people can't vote.
And cloudflare does not really have anything like that. Their customers yes, them no.
Like if some one wants to get a site down today so people can't vote.
And cloudflare does not really have anything like that. Their customers yes, them no.
> We would people try to do a ddos on them.
The same reason anyone ever DDOSes anything: because they're assholes.
The same reason anyone ever DDOSes anything: because they're assholes.
This is the most succinct and accurate assessment of anyone who DDOSSes anything.
They're a CDN for a sizeable chunk of the Internet. What do you think happens when you "temporarily" bring it to its knees?
It's not me! :)
There are a lot of people out there talking about the DDoS as well, some of them are talking about 400+ Gbps: https://twitter.com/search?q=ddos