SSH scans - I caught one(seclists.org)
seclists.org
SSH scans - I caught one
http://seclists.org/fulldisclosure/2010/Nov/228
68 comments
The correct way to address this is not to rely on fail2ban or start moving ports around (although these will remove noise from your logs, they shouldn't be solely relied upon) but to use public key authentication. It's not hard to set up and once you disable password authentication support on OpenSSH then the scans can try all they like, but they're not getting anywhere.
http://wiki.centos.org/HowTos/Network/SecuringSSH
http://wiki.centos.org/HowTos/Network/SecuringSSH
I have this problem right now.
A botnet found one of my computers. Most likely through IRC. They have been trying relentlessly to get my computer added.
They have no chance of getting in - using public key authentication - but it still takes quite a toll on my network speed.
That said, I have a file containing about 366 IPs belonging to this botnet, as well as a half-meg file containing nmap scans of all these IPs. What should I do with it?
A botnet found one of my computers. Most likely through IRC. They have been trying relentlessly to get my computer added.
They have no chance of getting in - using public key authentication - but it still takes quite a toll on my network speed.
That said, I have a file containing about 366 IPs belonging to this botnet, as well as a half-meg file containing nmap scans of all these IPs. What should I do with it?
With the places you access the computer from do you use dynamic or static IP addresses?
If they're all static IP addresses, try to block everything on port 22 except for your static IPs.
If it's dynamic you should have a pool address block, you can add the block in but there's a risk that if your block is in an ISPs home user pool that the botnet could infect people in your block.
If you can't use upstream filtering then I'd suggest you configure a firewall to do the same thing, either in hardware (preferable) or software.
If they're all static IP addresses, try to block everything on port 22 except for your static IPs.
If it's dynamic you should have a pool address block, you can add the block in but there's a risk that if your block is in an ISPs home user pool that the botnet could infect people in your block.
If you can't use upstream filtering then I'd suggest you configure a firewall to do the same thing, either in hardware (preferable) or software.
I have a somewhat unique situation which I think stops me from using this.
I access my servers from a shared (Windows) computer. I trust the other users enough not to install keyloggers, but not enough to put the private keys on the computer.
From some experimentation with putty, I can't find a reasonable workflow that lets me save the server configs, but prompts for the private key every time I log in, so I can load them (from a USB key for eg).
The best thing I've come up with is using long random, machine generated passwords with a password manager.
Any better ideas?
I access my servers from a shared (Windows) computer. I trust the other users enough not to install keyloggers, but not enough to put the private keys on the computer.
From some experimentation with putty, I can't find a reasonable workflow that lets me save the server configs, but prompts for the private key every time I log in, so I can load them (from a USB key for eg).
The best thing I've come up with is using long random, machine generated passwords with a password manager.
Any better ideas?
If you password protect the keys it's essentially the same at your end as if you were using a password. The only difference being that they need to copy the keys if after compromising the password they want to log on elsewhere.
A friend had me look at a server that had been compromised by SSH bruteforce a few months ago. The intruder was using it as an IRC bouncer, and he was a romanian named Alexino.
I actually found him on the IRC network, and he tried to get me to pay him to tell me how he got in :)
I actually found him on the IRC network, and he tried to get me to pay him to tell me how he got in :)
I've seen similar things come up before on HN and on mailing lists I'm on. Rather than keep repeating myself whenever this crops up, I've posted here: http://www.minklinks.com/weblog/2010/11/19/practical-guide-s...
book recommendation: http://www.amazon.com/Cuckoos-Egg-Clifford-Stoll/dp/06717268...
I blocked all of China and my logs decreased 65%.
There will always be the usual crowd of "conservatives-just-for-the-sake-of-being-conservative" crying out whenever this advice is given, but here goes: if possible in your environment, and for your users, just run your SSHd on a non-standard port, and the problem of automated scans will be a non-problem.
cd /var/tmp;
When I see things like this it makes me think that if standard paths weren't used, then it would it at least make things a little more interesting for the hacker. (They'd have to find a location first.)
When I see things like this it makes me think that if standard paths weren't used, then it would it at least make things a little more interesting for the hacker. (They'd have to find a location first.)
find / -maxdepth 3 -perm -7 -type d -printBut a find on root takes a lot of time in some cases, during which the attacker could be compromised. And the scripts might not use it for that reason. The script could use it as a backup only if the standard one doesn't exist, but still, it is a bump in the road, and might be worth it to cause the script to use find and delay it.
Hence the -maxdepth 3 param:
# time find / -maxdepth 3 -perm -7 -type d -print
/tmp
/var/tmp
real 0m0.034s
user 0m0.005s
sys 0m0.028s
It would be trivial to find a directory. You would be left with a bunch of boxes with things in random places, and programs possibly looking for non existent dirs.
But for simple devices or server setups with few changes over time, it might be worth it as it would at least cause a little more work to hack in. I remember that it used to be encouraged to use non-standard paths for security's sake, but then it seemed like too much work for the reason you stated. But these days, with more automated attacks relying on standard paths, I don't think it is out of the question.
like, "cd $TEMP;"?
Change the environment variable name, go through through your apps (and kernel probably) change the code to use a new one. Anything is doable, it just takes time. I'm not saying it is trivial by any means; I'm just saying it is an option. If you build everything from source, it is more doable. In fact, it would be nice to have a secure -nix distro that did it for you. (I know there isn't one that does this, but it could generate a random env var names and paths during install, and maybe change them on demand (by external boot).)
I'm perfectly aware that this could be "extremely difficult" or "not feasible" in many situations, but it wouldn't be so hard to control if you made the device and designed the OS for it (linux-based routers, etc.). I'm just tired of people giving up and laying things out like a holiday dinner because it's "too difficult". Good security requires sacrifice of work and time.
I'm perfectly aware that this could be "extremely difficult" or "not feasible" in many situations, but it wouldn't be so hard to control if you made the device and designed the OS for it (linux-based routers, etc.). I'm just tired of people giving up and laying things out like a holiday dinner because it's "too difficult". Good security requires sacrifice of work and time.
What you're describing is security through obscurity, not good security.
Anything that can be obfuscated programatically (while still remaining usable) can be rediscovered programatically. This accomplishes nothing.
Anything that can be obfuscated programatically (while still remaining usable) can be rediscovered programatically. This accomplishes nothing.
Interesting idea. Reading through the threads it becomes apparent that the attackers are targeting home wired/wifi routers. The attacker sits on your gateway and passively monitors your traffic with you being none the wiser!
How would you detect if your router was compromised?
Without having access to a box that was compromised it's hard to tell since the attacker could hide his tracks.
For this case specifically, if you're running openwrt you could try 'ps aux|grep syslgd' and see if you find anything. Then use 'lsof -p PID#' to see what files it's using. Or use 'lsof -i' to see what software has open connections. This can be very telling. Maybe 'netstat -anp| and looks for syslogd connecting out to the internet.
You should really follow basic security best practices and disable sshd on the outside interfaces, firewall unknown address, and switch sshd to a different port (security through obscurity will prevent the mass drive by port scanning).
For this case specifically, if you're running openwrt you could try 'ps aux|grep syslgd' and see if you find anything. Then use 'lsof -p PID#' to see what files it's using. Or use 'lsof -i' to see what software has open connections. This can be very telling. Maybe 'netstat -anp| and looks for syslogd connecting out to the internet.
You should really follow basic security best practices and disable sshd on the outside interfaces, firewall unknown address, and switch sshd to a different port (security through obscurity will prevent the mass drive by port scanning).
> and switch sshd to a different port
Big ups for this -- since moving SSH to port 8022 I get zero bruteforce attacks. Even blacklisting tools like fail2ban can't get that kind of result. Of course it can't be the only defense, but I've always configured my SSH daemons to be key login only (no passwords). Moving SSH just cuts down on the CPU cycles burned at rejecting drone scans.
Every time I come across posts like this I consider maybe setting up a honeypot but I'm not sure what I'd do with the results other than look over them every few months for my own amusement. Are there any honeypots that can automatically forward information about the attacks to a central location?
Big ups for this -- since moving SSH to port 8022 I get zero bruteforce attacks. Even blacklisting tools like fail2ban can't get that kind of result. Of course it can't be the only defense, but I've always configured my SSH daemons to be key login only (no passwords). Moving SSH just cuts down on the CPU cycles burned at rejecting drone scans.
Every time I come across posts like this I consider maybe setting up a honeypot but I'm not sure what I'd do with the results other than look over them every few months for my own amusement. Are there any honeypots that can automatically forward information about the attacks to a central location?
While it's a good idea to move ssh to a different port, if only to stop the annoying doorknob rattling in your log files, you should not pick an ssh port over 1024. Ports over 1024 are unprivileged ports, and if someone could somehow convince your ssh server to die, they could start up a counterfeit one and collect your password. A port under 1024 requires root access to be able to open it.
Of course, this only applies to linux systems where you don't trust local unprivileged users. Or software that you are running as an unprivileged user. So every system.
Of course, this only applies to linux systems where you don't trust local unprivileged users. Or software that you are running as an unprivileged user. So every system.
> Ports over 1024 are unprivileged ports
Good point, but
> they could start up a counterfeit one and collect your password.
you missed the part where I disable password logins on all of my boxes :-) The important point was that the system was already secure enough due to the key requirement, and moving the port was indeed just to stop the "doorknob rattling". If I suddenly find that a box I control is asking me for a password, I'm not going to just type my social security number in and hope for the best.
One could argue that using a port < 1024 makes it easier for the scanners to find, but frankly anything other than 22 (or a frequently scanned port) would work well enough.
Good point, but
> they could start up a counterfeit one and collect your password.
you missed the part where I disable password logins on all of my boxes :-) The important point was that the system was already secure enough due to the key requirement, and moving the port was indeed just to stop the "doorknob rattling". If I suddenly find that a box I control is asking me for a password, I'm not going to just type my social security number in and hope for the best.
One could argue that using a port < 1024 makes it easier for the scanners to find, but frankly anything other than 22 (or a frequently scanned port) would work well enough.
[deleted]
gxti seems to have been mildly aware of this, but I just set up ssh at home on 2222. So thanks for pointing this out.
Whatever happened to that port knocking idea a few years ago, did that go anywhere?
I had my internal monitoring system on a network I managed set up with port knocking. It ended up being more hassle than I thought it was worth, as it wasn't a box that was getting attacked much anyway.
It's really pretty easy to set up with iptables in Linux.
http://www.debian-administration.org/articles/268
It's really pretty easy to set up with iptables in Linux.
http://www.debian-administration.org/articles/268
Lots of people seem to recommend port knocking still today, but I personally won't use it as I'd hate to get behind a firewall (e.g. hotel, airport, university) where I can't send the right sequence of knocks for whatever reason (outbound ports firewalled).
I set it up a few months ago and ended up disabling it after a few weeks due to this reason. All my SSH servers have password authentication disabled and a strict AllowUsers list, anyways.
The only thing I do externally from sshd is have a cronscript run every 15 seconds to grep my authlog file to find any IP addresses that fail authentication and put them in a global blacklist for my PF firewall. It cuts down the ssh authlog noise considerably, and rarely accidentally blacklists myself.
The only thing I do externally from sshd is have a cronscript run every 15 seconds to grep my authlog file to find any IP addresses that fail authentication and put them in a global blacklist for my PF firewall. It cuts down the ssh authlog noise considerably, and rarely accidentally blacklists myself.
Don't forget `iptables -A INPUT -p tcp -m tcp --dport 22 -j TARPIT` ;)
Analyze the firmware. If it doesn't pass a checksum, it could be compromised. If the software doesn't pass a checksum, it's compromised. Strange traffic coming out of the router is a bad sign. If the router is running analysis on your packet, you will likely experience diminished throughput on the high end because of limited resources on the router.
I think if you can go without enabling remote access to a router, then you should. This will protect the router itself from direct login attacks from the outside, but if somebody compromises a machine on your LAN, you're boned. If you can't, and you have security concerns, you need to establish metrics of verifying whether or not the router is behaving the same way it was when it came out of the box and (presumably) was not compromised. Like I mentioned, software/firmware checksums are a possibility (these could be hard, though...getting a router to dump its internals out probably isn't a feasible solution for everything), but YMMV depending on what you do and what you need your router to do. My biggest guess at the tell-tale sign of a compromised router is that it just starts slowing down.
I think if you can go without enabling remote access to a router, then you should. This will protect the router itself from direct login attacks from the outside, but if somebody compromises a machine on your LAN, you're boned. If you can't, and you have security concerns, you need to establish metrics of verifying whether or not the router is behaving the same way it was when it came out of the box and (presumably) was not compromised. Like I mentioned, software/firmware checksums are a possibility (these could be hard, though...getting a router to dump its internals out probably isn't a feasible solution for everything), but YMMV depending on what you do and what you need your router to do. My biggest guess at the tell-tale sign of a compromised router is that it just starts slowing down.
Another "defense in depth" step would be to move your router off of 192.168.1.1. I recall seeing some browse-to-evil-webpage based attacks a while ago, and if the evil webpage can't guess your router IP (and it's not like it has access to your TCP settings in general) that can't work.
Obviously, just like moving SSH off of 22, this isn't a real defense, but it'll make it that much harder.
Obviously, just like moving SSH off of 22, this isn't a real defense, but it'll make it that much harder.
Firmware would not necessarily be modified in an attack like this, and this method would detect any persistent changes you make that are written back to NVRAM.
Definitely wise to prevent remote login from outside your LAN though. Also you can always run an IDS like Snort on your router; a proper IDS is probably the cleanest solution to this problem.
Definitely wise to prevent remote login from outside your LAN though. Also you can always run an IDS like Snort on your router; a proper IDS is probably the cleanest solution to this problem.
Depends on the compromise, but generally speaking the observable difference in behavior (assuming you are unable to monitor upstream of the router) is very minor. Greater power drain, and evidence of compromised information being used that could only have been gained from that source.
plan ahead & run an IDS on it
That's really quite sad - are the script kidiots hoping to take over a bunch of SGI's?
edit: I stand corrected..
edit: I stand corrected..
Looks like it tracks to a server/VPS place in The Netherlands
Wonder if he followed-up with the hosting service by reporting the address as being used in an attack. It would be interesting to turn the tables and listen in on some of his traffic going to that address.
Wonder if he followed-up with the hosting service by reporting the address as being used in an attack. It would be interesting to turn the tables and listen in on some of his traffic going to that address.
Anyone had much luck lately reporting to hosting companies/ISPs? I've reported 7 IPs in the last couple months (4 for malicious/bot activity, 3 for spambots), and never got any responses. Granted, they were all located in Hungary, Romania, or China.
As someone who runs a hosting company -- yes I fully investigate with the information given. Just please try to give as much as possible and make it easy for me to easily pinpoint things.
Oh, and please don't come with a snooty attitude, it's not pleasant to work with.
Oh, and please don't come with a snooty attitude, it's not pleasant to work with.
Do you respond to abuse reports? I've sent a bunch, and while I'd imagine at least some were actually followed up on, I've never received even one thank-you.
I'm not actually an ISP so my perspective is different but yeah I do, and I know the ISPs that I work with do too (Leaseweb, OVH). I have to work to keep things clean on my servers and to take reasonable steps to abide by their policies and inform my customers (although this kind of goes without saying).
One of the problems I imagine is size, I can still handle my couple of thousand customers and provide "good" support with enough time to investigate something that is described as a slowloris attack. I can open a dialogue with the customer and tell them that what they did was a little bit stupid and for the most part, the customer will understand and stop.
Although there was a time when I did investigate, couldn't find anything immediate (I'm spending say 5-10 minutes pruning a few TB of data here...) so the next point was to catch them in the act with a few iptables rules. Then the person who e-mailed me (and my host) pretty much turned hostile/gained a sense of entitlement and that was that. I wasn't much interesting in putting more time into it (so, didn't!). However this guy only produced a few text logs that could have been easily forged and I never heard more about it... who knows?
That also leads to another angle, how do you know who is telling the truth? The hosts are likely to side with the paying customer (well you never know with OVH) and there is little an outsider can do to prove their case in point except with easily fabricated text logs and events.
It's strange really, some of the people who I have followed up on have been very grateful (perhaps because they got their way), much in the same way going the extra mile for a customer might.
One of the problems I imagine is size, I can still handle my couple of thousand customers and provide "good" support with enough time to investigate something that is described as a slowloris attack. I can open a dialogue with the customer and tell them that what they did was a little bit stupid and for the most part, the customer will understand and stop.
Although there was a time when I did investigate, couldn't find anything immediate (I'm spending say 5-10 minutes pruning a few TB of data here...) so the next point was to catch them in the act with a few iptables rules. Then the person who e-mailed me (and my host) pretty much turned hostile/gained a sense of entitlement and that was that. I wasn't much interesting in putting more time into it (so, didn't!). However this guy only produced a few text logs that could have been easily forged and I never heard more about it... who knows?
That also leads to another angle, how do you know who is telling the truth? The hosts are likely to side with the paying customer (well you never know with OVH) and there is little an outsider can do to prove their case in point except with easily fabricated text logs and events.
It's strange really, some of the people who I have followed up on have been very grateful (perhaps because they got their way), much in the same way going the extra mile for a customer might.
I am impressed with your approach (and like the hosting business model you blokes seem to be employing at feralhosting.com).
Put you in my address book in case I ever develop for a UK-audience site.
Put you in my address book in case I ever develop for a UK-audience site.
I was the victim of an SSH scan once. I set up an old box as a dev server in my apartment for a project course in college. I was in a group of about 12 students. One of them whose username was robert set his password to robert (a brilliant move).
We only realized the machine was compromised because the interloper decided to pick two user accounts at random and delete them (another brilliant move).
Upon investigation I found that a keylogger had been installed in order to discover the root password. I inspected the output of the keylogger to trace the attacker's steps. Similar to the SSH scan in the article, the attacker had logged into his own FTP server to download various scripts and crackers. Well, the keylogger had logged his FTP password as well (whoops). Naturally I logged in and deleted absolutely everything in sight. :-P
We only realized the machine was compromised because the interloper decided to pick two user accounts at random and delete them (another brilliant move).
Upon investigation I found that a keylogger had been installed in order to discover the root password. I inspected the output of the keylogger to trace the attacker's steps. Similar to the SSH scan in the article, the attacker had logged into his own FTP server to download various scripts and crackers. Well, the keylogger had logged his FTP password as well (whoops). Naturally I logged in and deleted absolutely everything in sight. :-P
> Naturally I logged in and deleted absolutely everything in sight. :-P
...and you probably completely screwed over some hapless soul who was a victim of the same guy who cracked you. Do you think these guys put their IP address and FTP password out there for anyone to collect as evidence? Most likely, he hacked into an FTP server and hosted his malicious files there along with everything else that was already hosted there. Congratulations on breaking the law and ruining someone's day.
...and you probably completely screwed over some hapless soul who was a victim of the same guy who cracked you. Do you think these guys put their IP address and FTP password out there for anyone to collect as evidence? Most likely, he hacked into an FTP server and hosted his malicious files there along with everything else that was already hosted there. Congratulations on breaking the law and ruining someone's day.
I should have mentioned that I definitely thought of this scenario before I deleted anything and confirmed, via the hostname of the server, directory names, and file names, that everything on that server was for h4x0ring (at least for that user account).
I also thought of more subtle and clever ways to exact revenge, but decided I was too busy to pursue any of them.
Also: Do you think these guys put their IP address and FTP password out there for anyone to collect as evidence?
Yes, I do. You're giving the script kiddies far too much credit.
I also thought of more subtle and clever ways to exact revenge, but decided I was too busy to pursue any of them.
Also: Do you think these guys put their IP address and FTP password out there for anyone to collect as evidence?
Yes, I do. You're giving the script kiddies far too much credit.
I'm glad to hear that, thanks for clearing it up. Of course, I also intended my comment as a warning to others that the origination of hacking might not necessarily be the culprit.
Try to get the cops to care about this. Try to get campus security to care about this. Try to get your campus IT department to care about this. (They probably don't want you running servers anyway.)
Law enforcement has bigger issues. In a world where HBO can send a letter to my roommate from an IP in 10 days, and the police investigating a robbery have to wait 9 months for the same thing. You did the right thing.
I would have played a longer con game instead of nuking some random ftp server, he probably just set up another one.
Law enforcement has bigger issues. In a world where HBO can send a letter to my roommate from an IP in 10 days, and the police investigating a robbery have to wait 9 months for the same thing. You did the right thing.
I would have played a longer con game instead of nuking some random ftp server, he probably just set up another one.
Most likely the "hacker" broke into the FTP server because the admin of that server used the word "password" as his password. So natural selection took them both out. Two less idiots to deal with.
> Two less idiots to deal with.
That's an interesting attitude. Do you often find yourself having to "deal with" benign-but-ignorant FTP admins? Keep in mind that many people running FTP sites are not technical by trade, and you're using a rather wide brush stroke to paint a bad picture of many perfectly reasonable people.
Besides, judging by the recent submission of the decompiled code from RTM's worm from 1988 (http://news.ycombinator.com/item?id=1916186), hackers are a bit more sophisticated than you imply.
That's an interesting attitude. Do you often find yourself having to "deal with" benign-but-ignorant FTP admins? Keep in mind that many people running FTP sites are not technical by trade, and you're using a rather wide brush stroke to paint a bad picture of many perfectly reasonable people.
Besides, judging by the recent submission of the decompiled code from RTM's worm from 1988 (http://news.ycombinator.com/item?id=1916186), hackers are a bit more sophisticated than you imply.
One does not have to be "technical by trade" to know that using "password" as your password is wrong. Only idiots do that. And they do do that, otherwise the "sophisticated hackers" (I jest) would not be "cracking" all of these OpenSSH servers by guessing the passwords. The user sets passwords such as secret, letmein, abc123, etc. And then these "elite hackers" you speak of come along doing simple brute force because... wait for it... it works: http://www.dragonresearchgroup.org/insight/sshpwauth-cloud.h...
> using "password" as your password is wrong. Only idiots do that
and that means they deserve their files being deleted? What if that person was a non-tech savvy biologist working on cancer research? Or a college admin person and those files were the grades for the graduating year?
Serves 'em right for choosing a lame password and not setting up their own RAID server, eh?
and that means they deserve their files being deleted? What if that person was a non-tech savvy biologist working on cancer research? Or a college admin person and those files were the grades for the graduating year?
Serves 'em right for choosing a lame password and not setting up their own RAID server, eh?
I don't know jot all about car repair and if I tried to enact a casual repair and burnt myself or caused some other catastrophe, it would not be unreasonable to be considered an "idiot" by a professional mechanic.
The merit of the agent or situation involved (picking a cancer researcher seems like a clever and emotional rhetorical ploy) does not affect the demand for having domain knowledge when using or maintaining certain systems.
The merit of the agent or situation involved (picking a cancer researcher seems like a clever and emotional rhetorical ploy) does not affect the demand for having domain knowledge when using or maintaining certain systems.
If you run with scissors and fall, you may impale yourself. That does not mean (as you suggest) that they "deserved" it. It's a simple fact.
Except in this analogy, you tripped them.
This gentleman is illegally hosting my tools, which are copyrighted and not licensed for redistribution. Just because I try to pwn your box doesn't mean you have the right to violate my copyrights. Expect to hear from my lawyers.
Would you care to post a quick synopsis of what these tools are doing/their purpose? The thread mentions targeting routers - can you confirm?
I say, that's a joke son. </foghorn leghorn>
That's what my original thought was, but it didn't contribute anything to the discussion and wasn't really funny, or witty Given HNs population I had hoped you did author these tools. That you have so many votes is disheartening.
When it comes to investigating these sorts of thing, gray areas often pop up. Now, is fetching the payload yourself and hosting it for the world without the author's in this gray area? Probably not, definitely in the very light side of things if it is at all. It's not completely unworthy of thought and consideration however.
>wasn't really funny
Well, that's your opinion, and you are welcome to it. Do try not to be so puzzled that other people don't share your sense of humor though.
>wasn't really funny
Well, that's your opinion, and you are welcome to it. Do try not to be so puzzled that other people don't share your sense of humor though.
1. As far as I can tell, this specific attack is meant to target MIPS-based OpenWRT/DD-WRT devices, like the Linksys WRT series.
2. lsof and all that crap isn't available by default. So, use 'ps' and 'netstat -a', and 'ls -la /var/tmp' to poke around your router.
3. Go into the web admin interface and disable sshd on the WAN interface, if it isn't already (it's off by default). In DD-WRT, go to Administration->Management-> and ensure "SSH management" is disabled.