Observations from two weeks of SSH brute force attacks(lightbluetouchpaper.org)
lightbluetouchpaper.org
Observations from two weeks of SSH brute force attacks
http://www.lightbluetouchpaper.org/2012/01/25/observations-from-two-weeks-of-ssh-brute-force-attacks/
7 comments
It upsets me to hear this. I always wondered how Amazon handled abuse on EC2 as it's an insanely large service. At some point it has to become an issue? Does anyone have any experience with other types of abuse on EC2?
this has been a problem with AWS/EC2 for the last four/five years (since its creation): http://www.gossamer-threads.com/lists/nanog/users/105399
while that might be true, what's happening I'm quite sure can be proven to be malicious in court.
I don't like the idea that companies can turn a blind eye to illegal activity just cause they're getting a cut of the resulting profits.
I don't like the idea that companies can turn a blind eye to illegal activity just cause they're getting a cut of the resulting profits.
One of my linode servers was compromised and was used to originate such attacks. Linode proactively shut down the machine after a 24 hr notice period.
Unfortunately, I wasn't expecting this, so I missed their email and only realized the problem once my linode was offline.
I wouldn't be thrilled about either linode OR amazon
Unfortunately, I wasn't expecting this, so I missed their email and only realized the problem once my linode was offline.
I wouldn't be thrilled about either linode OR amazon
The Internet at large, on the other hand, would be quite happy with Linode in this particular instance.
Was there some other way they should have contacted you?
Was there some other way they should have contacted you?
That is always the most difficult part for me when trying to get these issues resolved - the legal side is complicated and I don't have the bankroll to get lawyers..
Just move your SSH off the port 22, hide it behind another instance and/or use fail2ban. 99% of problems with SSH scans/attacks sorted
Or just disable password-based authentication. These "attacks" are little more than log spam and are meant to prey on systems that host novice users with weakpasswords. Require PKI and the log spam goes away.
sure, I hope nobody is using password based authentication for SSH anyway :)
And I hope nobody is using key based authentication for SSH without storing the private part of their SSH key on an external smart card only, which is unlocked by an external hardware pin pad.
I was being pedantic of course. Password based SSH authentication is fine for 99% of the cases, as long as they use a "secure" password and don't leak it by writing it down or reusing it elsewhere.
I was being pedantic of course. Password based SSH authentication is fine for 99% of the cases, as long as they use a "secure" password and don't leak it by writing it down or reusing it elsewhere.
"as long as they..."
See, that's exactly why we don't allow password based authentication. While you can enforce strong passwords, you can't enforce that the user choose a password that is unique to your system, because you cannot know where else they may use this password.
The recent Dreamhost security disclosure is a great example. What are the chances that a contractor is using the same password for a Dreamhost shell account as they are my server? Pretty high, in my experience.
By disallowing password authentication, and requiring SSH-key auth, and requiring that contractors use a passphrase with their SSH-key, I have decreased the chances of a random break-in significantly.
See, that's exactly why we don't allow password based authentication. While you can enforce strong passwords, you can't enforce that the user choose a password that is unique to your system, because you cannot know where else they may use this password.
The recent Dreamhost security disclosure is a great example. What are the chances that a contractor is using the same password for a Dreamhost shell account as they are my server? Pretty high, in my experience.
By disallowing password authentication, and requiring SSH-key auth, and requiring that contractors use a passphrase with their SSH-key, I have decreased the chances of a random break-in significantly.
How do you "require" that they use a pass phrase with their key? You can tell them to do it, but it's up to them if they actually do.
I personally stand there with them (or over Skype) and ask them to show me how they manage their keys.
Most devs know how to use ssh-agent. Those who don't are really excited to learn that they can increase security without an increase in effort.
Password uniqueness, on the other hand, is something that requires continuous effort. There's an ongoing temptation to reuse the strong password that they've memorized for my server in other locations.
Like most things in security, it's a matter of layering and balance. You layer your security, and you stop at the point where the added effort doesn't justify the added security. I view the progression like this
OK: Strong passwords
Better: SSH-key authentication (much harder to brute force, not susceptible to disclosure through third-party security disclosure)
Best: SSH-key authentication with passphrase (prevents unauthorized use of private key if stolen)
Insane: SSH-key stored on external drive with imaginary super-encryption
Most devs know how to use ssh-agent. Those who don't are really excited to learn that they can increase security without an increase in effort.
Password uniqueness, on the other hand, is something that requires continuous effort. There's an ongoing temptation to reuse the strong password that they've memorized for my server in other locations.
Like most things in security, it's a matter of layering and balance. You layer your security, and you stop at the point where the added effort doesn't justify the added security. I view the progression like this
OK: Strong passwords
Better: SSH-key authentication (much harder to brute force, not susceptible to disclosure through third-party security disclosure)
Best: SSH-key authentication with passphrase (prevents unauthorized use of private key if stolen)
Insane: SSH-key stored on external drive with imaginary super-encryption
> Best: SSH-key authentication with passphrase (prevents unauthorized use of private key if stolen)
It doesn't prevent unauthorized use, it just makes it more difficult to attack with brute force. Once someone gets your private key, you should always assume it has been compromised, and replace it immediately. It's only a matter of time and resources before a determined thief has cracked your passphrase.
Edit: not to say I disagree with your recommendation that it's best, just pointing out that compromised private keys should not be taken lightly.
It doesn't prevent unauthorized use, it just makes it more difficult to attack with brute force. Once someone gets your private key, you should always assume it has been compromised, and replace it immediately. It's only a matter of time and resources before a determined thief has cracked your passphrase.
Edit: not to say I disagree with your recommendation that it's best, just pointing out that compromised private keys should not be taken lightly.
"external drive" is not what a smart card is. You can not read the key from the smart card. The actual crypto operations involved in performing the authentication are handled on the card it's self.
If your key is on a smart card it can not be stolen, even if the machine you're ssh'ing from has been compromised. This has real value and is not "insane"
In your setup, if their machine is compromised, it doesn't matter if they password protect the key, the attacker can key log the key password or replace the ssh executable with one which leaks the key, then ssh into your servers at will.
If your key is on a smart card it can not be stolen, even if the machine you're ssh'ing from has been compromised. This has real value and is not "insane"
In your setup, if their machine is compromised, it doesn't matter if they password protect the key, the attacker can key log the key password or replace the ssh executable with one which leaks the key, then ssh into your servers at will.
"If your key is on a smart card it can not be stolen, even if the machine you're ssh'ing from has been compromised."
If the machine you're ssh'ing from has been compromised by a halfway competent attacker, there's no need for them to get the key from the smart card. They can log in to the remote machine by simply piggybacking on to your ssh session (with a compromised ssh client).
If the machine you're ssh'ing from has been compromised by a halfway competent attacker, there's no need for them to get the key from the smart card. They can log in to the remote machine by simply piggybacking on to your ssh session (with a compromised ssh client).
Of course. There is still significant value in not allowing the attacker to ssh at will from any box at any time though.
Why, it's simple! They just send him their private key along with the passphrase, and he tests it to make sure it works!
(I've heard worse.)
(I've heard worse.)
For the sake of those who may not realise: You can take a password protected ssh key and remove it's password protection (if you know the password). And do the opposite of course (adding a password to a key without one)
You can do this with policy (or a custom SSH client) but it's really a mistake to treat it as the same situation unless your users are in the habit of sharing their private SSH key files around.
In most situations, getting access to the key file requires a local client compromise - at which point the attacker could simply install a keylogger.
In most situations, getting access to the key file requires a local client compromise - at which point the attacker could simply install a keylogger.
"Why of course I've picked a strong password, Mr. Sysadmin!" (password is "p4ssw0rd")
"Why of course I used a passphrase with my SSH key, Mr. Sysadmin!" (nope)
"Why of course I used a passphrase with my SSH key, Mr. Sysadmin!" (nope)
I'd still prefer ssh keys over a strong password. I don't know anyone who is perfect with regard to password sharing. Once you've memorized a wicked good password, there's a strong temptation to use it.
Say you've committed a really strong password to memory. One day, you find yourself setting up an account for some related service that you want to be super secure, but you're going to have to access it regularly. You decide to use your strong password. A year goes by and some hacker gains access to the said service's user database. Your super-strong password is now useless.
The great thing about ssh-keys is that the private key should be on your computer only. Generalized attacks don't go after single private keys, because you're only getting one set of credentials. If you're under focused attack, there are much easier attack vectors than trying to swipe a priv key from a dev's machine. If you can convince your dev to use passphrases on their ssh keys, you mitigate that attack vector as well.
Neither is perfectly enforceable, but using a passphrase with ssh-keys and ssh-agent is trivial. It's not perfect, but it's better than password auth.
Say you've committed a really strong password to memory. One day, you find yourself setting up an account for some related service that you want to be super secure, but you're going to have to access it regularly. You decide to use your strong password. A year goes by and some hacker gains access to the said service's user database. Your super-strong password is now useless.
The great thing about ssh-keys is that the private key should be on your computer only. Generalized attacks don't go after single private keys, because you're only getting one set of credentials. If you're under focused attack, there are much easier attack vectors than trying to swipe a priv key from a dev's machine. If you can convince your dev to use passphrases on their ssh keys, you mitigate that attack vector as well.
Neither is perfectly enforceable, but using a passphrase with ssh-keys and ssh-agent is trivial. It's not perfect, but it's better than password auth.
Whats wrong with writing it down on a sticky note taped to the box if the box is in a locked area?
No .. but it would damn cool if such hardware existed. Does it?
Our new national ID cards (called 'citizen card') here in Portugal work like that.
They have a keypair and the private part is never transmitted out of if: the card itself can (when plugged in a reader) sign and encrypt data sent to it. So it's secure even when using a public (not trusted) machine; I mean, it can spy on your connection, but it can't clone the key.
The certificate can be used in browsers to login to websites that support it or any other applications that implement the right PKCS standard.
They have a keypair and the private part is never transmitted out of if: the card itself can (when plugged in a reader) sign and encrypt data sent to it. So it's secure even when using a public (not trusted) machine; I mean, it can spy on your connection, but it can't clone the key.
The certificate can be used in browsers to login to websites that support it or any other applications that implement the right PKCS standard.
Yes it does. And it's pretty cheap too. I have an OpenPGP v2 smart card and an SPR-532 smart card reader. I will be writing up how I did it on my blog at https://grepular.com/blog/ probably next week. Need to find an hour or two to write the thing up.
I picked up an OpenPGP card and reader a few months ago and love the technology. My only beef is that the reliability of the software that talks to the smartcards is awful on the Mac.
Right now, it's definitely an approach to take if you are crypto nerd -- it's not ready for regular humans.
Right now, it's definitely an approach to take if you are crypto nerd -- it's not ready for regular humans.
Not to my knowledge, but similar approaches have existed for a long time now e.g. RSA SecurID [0]. One day I'm going to set up my Yubikey to perform a similar purpose, but my motivation levels correlate with my paranoia :)
[0] http://www.rsa.com/node.aspx?id=1156
[0] http://www.rsa.com/node.aspx?id=1156
I have a yubikey myself. The smart card method is much better though. When using a smart card, it is impossible even for the owner (who knows the access pin) to read the actual key. The crypto operations involved in the authentication take place on the card it's self. The smart card is protected by a pin, and will wipe it's self if the pin is entered incorrectly too many times. You can even use a smart card reader with a hardware pin pad built in so neither soft nor hard keyloggers can intercept it.
The weak point is you still don't know what you're signing, who or what you're authenticating, etc. if you don't have a trusted display and do the entire transaction on trusted hardware. (your host may be compromised, and could show arbitrary data, proxy connections, etc.) Protecting the key does go a long way, but even with a secure passphrase input and smartcard key operations, there is risk.
The ideal, IMO, would be some secondary hardware with a limited, secure OS which lets you put specific limits on how the key is used, showing a hash or identifying details of each transaction on the secure display, prompting you to verify each. E.g. to send a payment, the amount and payee (and date/sequence number) are shown on the secure display to verify before authorizing.
The ideal, IMO, would be some secondary hardware with a limited, secure OS which lets you put specific limits on how the key is used, showing a hash or identifying details of each transaction on the secure display, prompting you to verify each. E.g. to send a payment, the amount and payee (and date/sequence number) are shown on the secure display to verify before authorizing.
Is there an active exploit? Or are you just saying that this is the weakest form of auth for SSH?
password is the weakest form of authentication on open systems i.e. ones which you have to hand over to monkey grade users.
If you have a reasonable password policy yourself, I very much doubt it will cause you any problems.
I've been attacked by these for 10 years, probably totalling millions of attempts and I have a user name in their databases but due to a sensible password, nothing has got in.
I tend to use PKI and keyboard interactive logins though.
If you have a reasonable password policy yourself, I very much doubt it will cause you any problems.
I've been attacked by these for 10 years, probably totalling millions of attempts and I have a user name in their databases but due to a sensible password, nothing has got in.
I tend to use PKI and keyboard interactive logins though.
You could also set up your security group such that incoming 22 is denied by default, and open it up to individual IPs on-demand using a script like this:
MYIP=`curl http://automation.whatismyip.com/n09230945.asp`
ec2-authorize $SECGROUP -P tcp -p 22 -s $MYIP/32Or better yet, don't run SSH at all. Leave it to the pros.
I had an experience about a year ago on one of my home servers that convinced me of this. It was a Debian machine that I kept religiously up to date (daily cronjob and manual checking). It only had an external SSH port, I used log2ban, and disabled all logins except for one non-root user.
I got the common ssh brute force spam. Then one day I happened to be checking logs, and noticed rootkit logs in the /root directory. My server had been hacked! Fortunately whoever had broken in had completely botched it and left traces all over the system. It looked like they had given up at some point, despite having root access.
I later discovered that the attacker had used a 0-day openssh exploit that had been released that very morning. For that time period of 24-48 hours between release and fix, my server had been hacked. I only wonder how many other servers were hacked by skilled people that didn't leave evidence behind.
So morale of the story - you can't secure anything that is publicly facing on the internet. And if you really need to do it, it requires constant vigilance, DMZs, and all that other stuff that is not worth it unless you're being paid.
I had an experience about a year ago on one of my home servers that convinced me of this. It was a Debian machine that I kept religiously up to date (daily cronjob and manual checking). It only had an external SSH port, I used log2ban, and disabled all logins except for one non-root user.
I got the common ssh brute force spam. Then one day I happened to be checking logs, and noticed rootkit logs in the /root directory. My server had been hacked! Fortunately whoever had broken in had completely botched it and left traces all over the system. It looked like they had given up at some point, despite having root access.
I later discovered that the attacker had used a 0-day openssh exploit that had been released that very morning. For that time period of 24-48 hours between release and fix, my server had been hacked. I only wonder how many other servers were hacked by skilled people that didn't leave evidence behind.
So morale of the story - you can't secure anything that is publicly facing on the internet. And if you really need to do it, it requires constant vigilance, DMZs, and all that other stuff that is not worth it unless you're being paid.
What exploit was it specifically?
In your specific case, this wouldn't have happened if your SSHd had listened to something else than port 22.
s/wouldn't have happened/would've been much less likely/
I agree - I did this a long time ago; while it's fun to go through logs sometimes, I prefer to weed-out the general bombardment. My hits dropped from about 90 a day to 0 (after 6 months).
A long time ago I used to have a general purpose server on a DSL line with port 22 open. I thought everything was fairly secure, patched up, etc... I never thought brute forcers would be able to guess one of my accounts (which was a friend's account actually). The kicker? The password was the username. I found out about the rootkit a few days after they got in. I tell ya, kids these days just don't know how to hide a backdoor.
A really good port knocker is your best defense against rogue attackers of an external service (because there is no defense against a 0-day). For an iptables-supported system I recommend Knockknock by Moxie Marlinspike (http://www.thoughtcrime.org/software/knockknock/).
A really good port knocker is your best defense against rogue attackers of an external service (because there is no defense against a 0-day). For an iptables-supported system I recommend Knockknock by Moxie Marlinspike (http://www.thoughtcrime.org/software/knockknock/).
For brute forcing, without resorting to port knocking, fail2ban works well and has a pluggable system to handle more than ssh.
Does fail2ban work against brute force distributed attacks?
I've noticed a few attacks that seem to be orchestrated from the same people but using different IPs.
I've noticed a few attacks that seem to be orchestrated from the same people but using different IPs.
It doesn't specifically target distributed attacks, but I'm not really sure how something like that could know whether incoming requests are part of a botnet or ligitimate logins.
Or use only public-key authentication. I've run many internet-exposed servers for years without problems or kludgy obscurity measures like port knocking simply by disabling all forms of password authentication. This is faster, easier and has the advantages of not requiring non-standard client software, being observable by an attacker on the same network, and avoids the sort of rote password entry which causes mistakes and discourages long, complex passwords.
With public key only authentication, how does a user get their public key on to the server in the first place?
I ask because I have two servers, one I am a client at DreamHost and I can't configure their sshd daemon, the second is my own VPS where I am root.
I ask because I have two servers, one I am a client at DreamHost and I can't configure their sshd daemon, the second is my own VPS where I am root.
You only need the ability to write to a file in your home directory. In the case of a VPS you could do that before editing the sshd config (in large environments this is trivially automated as part of your install process, which is why e.g. Ubuntu EC2 images ship with public-key-only SSH by default).
In the case of a shared server you can't disable password auth, which is a powerful argument for using a unique password since that system is far more likely to be compromised.
In the case of a shared server you can't disable password auth, which is a powerful argument for using a unique password since that system is far more likely to be compromised.
I think that a more reasonable explanation could be that the attackers were careless about building the dictionary. If it's built by parsing files and you feed the wrong file or the parser does not work correctly you end up with a dictionary with lots of bogus entries. That seems a simpler explanation than:
"The best guess is that these passwords were collected from an unhashed password database, or from a trojaned SSH server or client."
or
"This might be due to the brute force tool not properly interpreting comments in the dictionary file, or the attacker not understanding the comment notation"
"The best guess is that these passwords were collected from an unhashed password database, or from a trojaned SSH server or client."
or
"This might be due to the brute force tool not properly interpreting comments in the dictionary file, or the attacker not understanding the comment notation"
Interesting idea. Assuming competence on the part of your attacker isn't as dangerous as assuming incompetence, but it can be just as mistaken.
It could be more haste than incompetence. Anyway, if you think about massive brute force attacks like this, they are 'silly' easy no-brain attacks more likely to come from script kiddies who downloaded a tool someone else wrote than real attackers.
Heh, those passwords are in Romanian
TiganilAFloriNTeleormaN - Gypsies at Florin in Teleorman
Fum4tulP0@t3Uc1d3R4uD3T0t - Smoking can kill awfully bad
TiganilAFloriNTeleormaN - Gypsies at Florin in Teleorman
Fum4tulP0@t3Uc1d3R4uD3T0t - Smoking can kill awfully bad
What is with Romanians? Most of the metadata sent (see at the bottom) is in Romanian.
However, I decided to run an additional experiment to contact Amazon since the IP was originating from an EC2 instance. Amazon contacted me after I filed an abuse report and said they were investigating.
A week goes by and I'm still getting hammered. So I email Amazon and asked when it will be resolved. No response. So I email again, again, and again. I finally get a response saying they have resolved it, but I'm still getting hammered from the same IP address.
So I email them once more asking - what is it exactly you have resolved?
No response.
I presume it is not in Amazon's best interest to resolve such issues as long as people are paying for their instances....