GitHub: About This Week's Availability(github.com)
github.com
GitHub: About This Week's Availability
https://github.com/blog/1036-about-this-week-s-availability
10 comments
I remember reading a comment a long time ago -- maybe here or Quora, I think -- talking about how people don't usually realize how much trouble organized crime gives small and midsize businesses. You'll start getting DDOSed and then an email appears a day later: pay us and we'll stop bringing down your site. I have no idea if extortion is what's happening here versus simple lulzseekers or a grudge someone holds against them, but I wouldn't be surprised.
"Organized crime" or just regular crime?
In some countries that are absolutely poor, it is entirely common to have the local mob recruiter look for young talented kids that they can bring into their organization. They make them an offer when the kids are out begging for change or something like "we'll take care of your mom and make sure she doesn't break her legs if you come and get an education in our school, we might even pay you."
The youngster ends up in basically a run-down warehouse full of other kids each taught how to look for basic exploits like mysql injection or use root kits, and they sit around and do this all day.
This is why some countries are typically COEs for hacking, such as Russia with crimeware kits, and until recently Brazil for bank hacking.
This is also why you need coordinated efforts of special agents armed with assault rifles to clean out some spam-mills sometimes (they're taking them away from some mob).
What I've described is a bit outmoded, typically this behavior is turning into other things such as this: http://abh-news.com/cybercrime-china-hackers-training-camp-c... as well as the makers of exploit scripts are just getting better and recruiting these kids doesn't make sense any more... you can intelligently scan for millions of sites using phpmyadmin and automatically run your exploit with it.
As with any crime you find the trends all over the map: not all robberies are mob related, not all of them aren't, some of them are more elaborate than others, etc.
The youngster ends up in basically a run-down warehouse full of other kids each taught how to look for basic exploits like mysql injection or use root kits, and they sit around and do this all day.
This is why some countries are typically COEs for hacking, such as Russia with crimeware kits, and until recently Brazil for bank hacking.
This is also why you need coordinated efforts of special agents armed with assault rifles to clean out some spam-mills sometimes (they're taking them away from some mob).
What I've described is a bit outmoded, typically this behavior is turning into other things such as this: http://abh-news.com/cybercrime-china-hackers-training-camp-c... as well as the makers of exploit scripts are just getting better and recruiting these kids doesn't make sense any more... you can intelligently scan for millions of sites using phpmyadmin and automatically run your exploit with it.
As with any crime you find the trends all over the map: not all robberies are mob related, not all of them aren't, some of them are more elaborate than others, etc.
As I understand it, eastern european criminal organizations (I hesitate to use the term "mafia") control pretty big botnets, and use them to do exactly this.
Although github is sort of an odd target for them to be going after. Maybe their hired geeks are just testing their own software - github seems like the sort of place that would figure out how to defeat a DDoS attack, and by learning how, the black hats could improve their tools.
Although github is sort of an odd target for them to be going after. Maybe their hired geeks are just testing their own software - github seems like the sort of place that would figure out how to defeat a DDoS attack, and by learning how, the black hats could improve their tools.
Are there any good guides on how to combat a DDOS? I'm curious how a team of good technical people without past experience in the area could react.
Fighting a DDOS is very hard if you do not have the resources to purchase more bandwidth. The DDOS attacks I've dealt with have been almost exclusively UDP-floods in the 500-2000 mbit range. No amount of server-side configuration will fix that unfortunately. If the attack is some more creative form, but with less bandwidth (SYN flood etc.) a cisco router can help out a lot.
Your best bet is to move to a protected hosting provider (dragonara or ddoshostingsolutions are quite good for Europe / US). Failing that you can purchase a bunch of VPS servers, round-robin DNS to them and reverse proxy to your actual server (and keep that IP off the DNS system entirely). This way only the proxy servers will go down if you are attacked.
Edit: If they are not spoofing they sender address, you can achieve some success by mailing the hosting provider of the IP performing the attack. In 90% of cases the server has been hijacked and they will shut it down, this only works for US / European companies though since Chinese / Russian hosting providers never ever reply in my experience. For maximum success you can attach a tcpdump of the attack traffic.
Your best bet is to move to a protected hosting provider (dragonara or ddoshostingsolutions are quite good for Europe / US). Failing that you can purchase a bunch of VPS servers, round-robin DNS to them and reverse proxy to your actual server (and keep that IP off the DNS system entirely). This way only the proxy servers will go down if you are attacked.
Edit: If they are not spoofing they sender address, you can achieve some success by mailing the hosting provider of the IP performing the attack. In 90% of cases the server has been hijacked and they will shut it down, this only works for US / European companies though since Chinese / Russian hosting providers never ever reply in my experience. For maximum success you can attach a tcpdump of the attack traffic.
I was poking around a botnet today and saw that the particular one I was looking at was just DDOSing DNS (port 53). Could you just use an /etc/hosts file and turn off UDP entirely? My understanding is that part of the problem is that the reason 53 is so vulnerable is because the OS sits there waiting for any UDP replies from DNS on it.
If your server is running slow because of the load, and not because of the bandwidth of the attack that might be a viable route. It is trivial to use iptables to block all UDP traffic except from your configured DNS server.
Most of the cases though it's as simple as attack is 200 mbit and connection is 100mbit -> no amount of dropping packets once they reach the server will do anything, as the connection is entirely saturated.
Most of the cases though it's as simple as attack is 200 mbit and connection is 100mbit -> no amount of dropping packets once they reach the server will do anything, as the connection is entirely saturated.
You can filter IPs on country given the correct software. During an attack, just prioritise the traffic from high use countries. It might work?
Who would DDOS GitHub and for that matter, why?
extortionists.
There's a growing band of pirates (the web kind) who hold sites to ransom via their massive botnets, in exchange for payment -- 'protection money'.
There's a growing band of pirates (the web kind) who hold sites to ransom via their massive botnets, in exchange for payment -- 'protection money'.
But why Github? Wouldn't a less tech-savvy business be much more likely to cave in, and have a better cash flow? (e.g., porn and gambling sites?)
Uptime is very valuable to github. For example, I can't easily deploy to my site when githubs down. I suspect many others have bought into its availability too.
I had a very slow deploy earlier this week; if it had gotten canceled in the middle of it that would have been pretty bad. Maybe the deployment script should copy the repo head to one of my own servers and continue from there? What do people do?
Sure, which is why you attack them too.
Random thoughts but:
- Github are obviously very successful and may have money - their clients are reacting quickly on twitter so attackers might think they have an edge in terms of pressuring
- Github are obviously very successful and may have money - their clients are reacting quickly on twitter so attackers might think they have an edge in terms of pressuring
Not in that line of business but I'd expect any porn and gambling site with half-decent cash flow to be quite tech-savvy when it comes to securing it from all the DDOSers, fraudsters and other hostile entities they have to deal with daily.
Do any of these extortionists have a reputation for actually stopping the attack once they get paid? I certainly wouldn't trust some random group that's currently DDOSing me enough to pay them.
It reminds me of the Vikings. They would plunder throughout Europe including modern day France. It got so bad that the coasts ended up practically abandoned.
The rich and weak Frank empire, a bad combination, resorted to the disastrous policy of bribing the invaders to leave. It only served to mark the Franks as a good target and emptying the treasury. The Vikings would leave, plunder some other place and come back for more.
In the end, the Franks ended up giving part of their land on the coast : Normandy. This served to settle the invaders down while making THEM have to protect their newly acquired territory against other vikings.
The rich and weak Frank empire, a bad combination, resorted to the disastrous policy of bribing the invaders to leave. It only served to mark the Franks as a good target and emptying the treasury. The Vikings would leave, plunder some other place and come back for more.
In the end, the Franks ended up giving part of their land on the coast : Normandy. This served to settle the invaders down while making THEM have to protect their newly acquired territory against other vikings.
Wouldn't it be in their favor to stop? No one would pay them if they had a reputation of continuing the attack after being paid. It also seems like it'd be a waste of their resources to keep going.
Unless they changed their mind and decide since you paid so easily, you might have a little more. There's no reputation to maintain in this activity.
Fool me once, shame on you. Why would they pay out again, if the attackers have made it clear that there's no reward for paying?
Would you want to have an identifiable reputation if you're DDoSing people?
Your logic is sound - why trust a criminal?
But it's not like there's a better business bureau from organized crime.
Your logic is sound - why trust a criminal?
But it's not like there's a better business bureau from organized crime.
My thoughts exactly. I'm inclined to believe with another commenter that they just picked GitHub as testing grounds.
Especially depending on the sum of money demanded, I can imagine it being cheaper to either spin up more servers to handle the load, or hire a DDoS specialist to help you out, or make it someone else's problem by moving behind a thirdparty CDN.
Especially depending on the sum of money demanded, I can imagine it being cheaper to either spin up more servers to handle the load, or hire a DDoS specialist to help you out, or make it someone else's problem by moving behind a thirdparty CDN.
or just bored kiddies, happens quite a lot
Besides reasons mentioned, it could benefit their competitors. Remember that sourceforge was under heavy ddos not even a year ago.
I'm not saying 37signals did it, but I bet someone who read this post did: http://37signals.com/svn/posts/3099-benchmarking-basecamps-u...
I'm sure it was tempting more than a few people with the resources.
Edit: just to be clear I meant any reader of signal vs. noise could have seen that (and Jesse's #1 comment and the jokes about DDOS in the comments) and decided to pull a prank. 37signals would never do that to anybody.
I'm sure it was tempting more than a few people with the resources.
Edit: just to be clear I meant any reader of signal vs. noise could have seen that (and Jesse's #1 comment and the jokes about DDOS in the comments) and decided to pull a prank. 37signals would never do that to anybody.
My wife is pretty convinced I brought this upon myself with my comment on that post. Lol, you never know.
This seems a bit far fetched.
I guess I didn't display my 37signals fanboy card prominently enough.
If I were a DDoS attacker, I wouldn't want to make enemies with the sort of people that rely on github.
Though it's easy to imagine that those guys think differently than I do.
Though it's easy to imagine that those guys think differently than I do.
Very nice. Just one thing: I already know I'm awesome and I don't need my source revision control provider to boost my self confidence (I'm referring to the line "...you, our awesome GitHub users"). OTOH, it's clear that it's GitHub's employees, rather than its users that need constant reassurance of their coolness. GitHub is a nice company that provides a good, solid and necessary service - hosted SCM. Then why is it that their company blog focuses on their drinking habits?
Dear GitHub,
Although your product may sound boring to laymen, you provide a good service to a very important industry. Your "boring" day job reflects nothing on your personal and very exciting lives. Your customers, however, like you for what you provide them with, not for your companionship, and I'm sure your friends like you for the opposite reasons, as they very well should. In fact, everybody loves you for many reasons, and you are all very lovable. So please, keep your extra-curricular activities to your friends, and your work activities to your customers. You can call your friends "awesome" if that's your kind of thing, but for various reasons it is better to treat your customers with proper decorum. If you're unable to keep the two separate, you're in for some bitter disappointment later in life.
Love, everyone.
Dear GitHub,
Although your product may sound boring to laymen, you provide a good service to a very important industry. Your "boring" day job reflects nothing on your personal and very exciting lives. Your customers, however, like you for what you provide them with, not for your companionship, and I'm sure your friends like you for the opposite reasons, as they very well should. In fact, everybody loves you for many reasons, and you are all very lovable. So please, keep your extra-curricular activities to your friends, and your work activities to your customers. You can call your friends "awesome" if that's your kind of thing, but for various reasons it is better to treat your customers with proper decorum. If you're unable to keep the two separate, you're in for some bitter disappointment later in life.
Love, everyone.
I've used Github frequently all week,personally and professionally, and if they temporarily lost a nine or three, I haven't noticed.
Couldn't they just keep adding IP's to some blacklist (not htaccess, but something more efficient), and the attackers will eventually run out of IPs to attack from? There's only a finite # of computers you can really have control of. So you end up blocking some innocents, but you take care of that after the attack is over.
If it is a smart DDoS the requests won't look any different than typical users visiting your site.
Maybe a whitelist could work. My IP has visited in the past week / I have a legit session cookie, now gimme bandwidth.
This only works if you decide you want to never accept a new customer again.
Not necessarily, though it depends on the complexity you want to throw at the problem.
If you have known (and proven) legitimate traffic, give it a high QoS.
For never-been-seen IPs, provide a limited initial rate, with training based on experience.
Use tools such as the ASN Routeviews project to identify contiguous blocks of IP space (or any other source of BGP routing data, but the RV data are queryable via DNS and downloadable as zonefiles).
ID bad actors and either block or severely limit them.
Train up or down other traffic as appropriate.
This leaves you tracking large amounts of IP space. Doable in IPv4, somewhat more difficult under IPv6, though you're still probably going to be able to do something reasonable.
I'd like to see more of this pushed into the routing/networking layer, automatically, based on application-layer-based feedback. Maybe someday.
If you have known (and proven) legitimate traffic, give it a high QoS.
For never-been-seen IPs, provide a limited initial rate, with training based on experience.
Use tools such as the ASN Routeviews project to identify contiguous blocks of IP space (or any other source of BGP routing data, but the RV data are queryable via DNS and downloadable as zonefiles).
ID bad actors and either block or severely limit them.
Train up or down other traffic as appropriate.
This leaves you tracking large amounts of IP space. Doable in IPv4, somewhat more difficult under IPv6, though you're still probably going to be able to do something reasonable.
I'd like to see more of this pushed into the routing/networking layer, automatically, based on application-layer-based feedback. Maybe someday.
This is only to block the attack. The thinking is when the attackers run out of PC's they will give up. You lose some customers in the short-term. If you don't do any blocking, they will just keep attacking and attacking.
I thought that botnet operators had methods of rapidly changing bots' IPs so as to defeat IP blocks.
The attacker is most likely using a program that spoofs the sender IP so every packet has a unique IP. It's not feasible to block that. You need a more intelligent router to block attacks of that kind (and lots of bandwidth).
I wonder if something like http://www.cloudflare.com could help.
I think if you build your app properly the only problem with DDoS is traffic. Github partnerships with Rackspace so their cloud traffic can't be expensive. Correct me if I'm wrong.
I suspect you may be oversimplifying the solution or undersimplifying the nature of these kinds of attacks. What would constitute a 'properly' infallible defense to you?
Github is really great, but I'm a cynic.
Indeed, why _would_ anyone want to DDoS github? How can we believe Github that the outages were due to a DDoS? They're smart people, aren't they using solid load balancesrs that can mitigate DDoS attacks? Why haven't they issued an actual statement describing the supposed attacks in better detail?
Indeed, why _would_ anyone want to DDoS github? How can we believe Github that the outages were due to a DDoS? They're smart people, aren't they using solid load balancesrs that can mitigate DDoS attacks? Why haven't they issued an actual statement describing the supposed attacks in better detail?
Load balancers generally don't offer as much mitigation against DDoS as you think. The load balancer will probably fall over if the backend servers don't do so first. The sheer volume of seemingly valid traffic is what kills you.
Which is why effective anti-DDoS means working with your upstream provider to figure out how to differentiate between real traffic and fake traffic. The best solution to a DDoS is the phone # of a tech that can implement firewall rules upstream and a good traffic analyzer to tell them exactly how to filter it.
Which is why effective anti-DDoS means working with your upstream provider to figure out how to differentiate between real traffic and fake traffic. The best solution to a DDoS is the phone # of a tech that can implement firewall rules upstream and a good traffic analyzer to tell them exactly how to filter it.
Often you can't pick apart the good traffic from the bad.
Fortunately bots are usually pretty stupid. If you can outrun them on bandwidth, then change /victimpage.html to 302 to /victimpage-new.html. The web server or load balancer can send those redirects really fast and it doesn't take much bandwidth either. I have never seen a bot chase that redirect.
After a particularly nasty DDOS attack (where our upstream provider just shrugged their shoulders) I wrote an F5 iRule:
1. Check for the IAMNOTABOT cookie 2. If not there, redirect to /cookie-me?oldpage=the_page_you_were_trying_to_access 3. Set IAMNOTABOT=true cookie 4. Redirect to the old page
Fortunately bots are usually pretty stupid. If you can outrun them on bandwidth, then change /victimpage.html to 302 to /victimpage-new.html. The web server or load balancer can send those redirects really fast and it doesn't take much bandwidth either. I have never seen a bot chase that redirect.
After a particularly nasty DDOS attack (where our upstream provider just shrugged their shoulders) I wrote an F5 iRule:
1. Check for the IAMNOTABOT cookie 2. If not there, redirect to /cookie-me?oldpage=the_page_you_were_trying_to_access 3. Set IAMNOTABOT=true cookie 4. Redirect to the old page
Another, similar, technique is checking for a NOBOT cookie and if it doesn't exist, serve up a page that uses javascript to set the cookie, then reloads.
This requires the bot to interpret javascript, and it can easily be configured in nginx and other load balancers.
This requires the bot to interpret javascript, and it can easily be configured in nginx and other load balancers.
Ask anyone who's been there and they'll tell you that load balancers only make DDoS attacks harder. Unless you have something like AWS AutoScaling, they'll just generate more traffic - and if you do, it would quickly become a financial DDoS. That's why sites like Yahoo.com have been taken out: you have finite server capacity but the attacker merely has to find another another exploitable system somewhere on the Internet.
The way you deal with a DDoS is hard: you can try to identify distinguishing characteristics of the attack so traffic can be blocked as far out as possible and you try to figure out who's behind the attack so you can get law-enforcement involved. Otherwise you're just playing whack-a-mole with compromised home computers all around the globe…
The way you deal with a DDoS is hard: you can try to identify distinguishing characteristics of the attack so traffic can be blocked as far out as possible and you try to figure out who's behind the attack so you can get law-enforcement involved. Otherwise you're just playing whack-a-mole with compromised home computers all around the globe…
As another poster mentioned, DDOS attacks on successful companies are not uncommon. And there isn't an easy fix for this.
That's not just cynical; that's down right paranoid.
> Indeed, why _would_ anyone want to DDoS github?
"Some people just want to watch the world burn."
Organized crime extorting sites with DDoSes is certainly not unheard of, either.
"Some people just want to watch the world burn."
Organized crime extorting sites with DDoSes is certainly not unheard of, either.
Responding to your last sentence, there was a comment on the blog post that they are being vague to not give anything away to the attackers. If they start describing the attack (even how they perceive it) and/or mitigation steps, they are giving targeting data to the people doing the attack.
Arguably, if your load balancer setup is such that traffic can increase 10000x from the minute before and nothing falls over, you were probably incredibly over-provisioned in the first place.
Unless your regular traffic pattern has 10000x increases built into it, why would you spend all the extra expense to be able to handle that level of load?
Unless your regular traffic pattern has 10000x increases built into it, why would you spend all the extra expense to be able to handle that level of load?