Protecting sshd using spiped(daemonology.net)
daemonology.net
Protecting sshd using spiped
http://www.daemonology.net/blog/2012-08-30-protecting-sshd-using-spiped.html
8 comments
I covered this in a blog post a few years ago -- probably best if I point you at it rather than trying to cover everything here: http://www.daemonology.net/blog/2009-06-24-encrypt-then-mac....
Wow. That was more than I hoped for. I especially appreciated the closing paragraph. Thanks for taking the time to write it and to point it out here.
However, one of the points you made as an argument against authenticated encryption modes is actually a point for it, unless I'm not getting something. The point I'm referring to is that people write bad code and that every line of code includes a potential security vulnerability. Every time someone uses CTR+MAC, they're writing code to glue those together, and there are opportunities for missteps. Furthermore, that code that is viewed by a small number of people. By using an authenitcated encrytion mode, one can reduce the lines of code you're writing because it the functionality is provided by the implementation, which will certainly have been looked at more times than one's own project.
[Edit]
Maybe this is what I was missing: "Encrypt-then-MAC is essentially impossible to get cryptologically wrong."
However, one of the points you made as an argument against authenticated encryption modes is actually a point for it, unless I'm not getting something. The point I'm referring to is that people write bad code and that every line of code includes a potential security vulnerability. Every time someone uses CTR+MAC, they're writing code to glue those together, and there are opportunities for missteps. Furthermore, that code that is viewed by a small number of people. By using an authenitcated encrytion mode, one can reduce the lines of code you're writing because it the functionality is provided by the implementation, which will certainly have been looked at more times than one's own project.
[Edit]
Maybe this is what I was missing: "Encrypt-then-MAC is essentially impossible to get cryptologically wrong."
Oh my gosh buddy if you're going to say something that interesting, please explain it for the rest of us with a few extra sentences what you're asking. I'm sure the author will know instantly, but bring the rest of us up with an extra nugget because the tone of your question sounds fascinating and if you double it out, I bet it will be.
I downvoted, and I'd like to explain why and with the disclaimer I also did not understand a lot of what was asked. This was a technical question on a subject that takes years to master properly. It's not fair on the questioner to have to try and explain that to the rest of us when Google will suffice, nor on the author that would understand the question as asked to have to waste time getting to the point of the question.
That's fine. I took his question as obtusely academic and that he would have been the best person to easily append a few clarifying statements as to why the rest of us should have to process his question. And I understand that I am being downvoted by the people that either understand his question and sympathize, or by people that feel I should have devoted some time to understanding his question. I do not agree with this, and so my original decision to ask him for more. I find that I generally have to emulate a different persona than I really am in order to stay above the ice around here. I don't take your downvote personally. [This was written before the grandparent edited their post with lots of clarifying information]
Here is a better idea. If it looks like a duck, walks like a duck and quacks like a duck, is it a duck? Run ssh on port 443 at the same time you run a web server on the same port. http://www.rutschle.net/tech/sslh.shtml I've seen implementatios in about a page of C, which I can vet.
That is terrifyingly evil. And I mean that in the best possible way.
As someone who doesn't have a very subtle understanding in English, I can't figure out if this is an endorsement, or very much not one? (serious question! I'd rather fly by your crypto-intuitions than my own...)
It's incredibly cool. Not something I'd necessarily want to actually use in production, though.
The concept that the protocol multiplexer uses is pretty sound and simple. I have and would continue to deploy in production, but as usual you have to know what is going on and what the implications are. You do pay a known penalty in complexity and startup time but you gain a world of obscurity. It's like every other tool; it has it's uses, but it's not suitable for all problems.
The fact that an implementation is possible in a few pages of C goes a long way, especially when the install base for that tool is small and you need to understand the details. If nothing else, the concept stands on it's own and might offer ideas in solving other problems.
The fact that an implementation is possible in a few pages of C goes a long way, especially when the install base for that tool is small and you need to understand the details. If nothing else, the concept stands on it's own and might offer ideas in solving other problems.
In the ssh protocol, the server sends a banner first. For HTTPS, the client sends certificate data first. How is it possible to run these 2 services on the same port without changing the client?
The link he gave is a custom multiplexing daemon that handles those cases. A timeout value is used to connect to the SSH daemon if the server hasn't received anything from the client.
I've done the same thing to set up a "SVN-or-git repository" on a machine on which I only had my own shell account. Fortunately, that never made it to production.
I can't help but notice that this tool was created to circumvent private networks. If you just wanted to write a crypto tool, kudos, but this is not a good idea for production data. I also recommend focusing on system hardening versus throwing crypto and fewer lines of code at the problem. The net benefit is ten times that of a more complex set-up that only protects a corner case.
Take your original example: 17 servers and you want to share data between them on the internet. The most effective way to do this is to put them all on one or two hosting providers and use private IPs to connect them all via the same backend. This way there is no surface to attack (if the hoster VLAN'd your hosts correctly). Back in the day we used to do this by putting a private switch in a rack that nothing else could touch, but it seems the days of private cage monkeying has disappeared.
The other way is by using a VPN. A VPN is simply a network interconnect. Based on some authentication data, allow a host to connect to a private network. If done properly, there is no service to take advantage of and the daemon can run completely without any capabilities or system access, save of course network traffic. You then add firewalling to ensure only the right hosts get to the right services. You can even make this a single stand-alone VPS host to add granularity and separation of privilege, not to mention reduced attack surface.
Your tool is theoretically simpler in practice than either of those methods (I don't find setting up a VPN to be complex, but others might). But it comes with the same risks as OpenSSH, save that (I imagine?) you could set it up to be network-access-only like a VPN. The difference for a VPN being you wouldn't need to configure multiple instances of an application.
Take your original example: 17 servers and you want to share data between them on the internet. The most effective way to do this is to put them all on one or two hosting providers and use private IPs to connect them all via the same backend. This way there is no surface to attack (if the hoster VLAN'd your hosts correctly). Back in the day we used to do this by putting a private switch in a rack that nothing else could touch, but it seems the days of private cage monkeying has disappeared.
The other way is by using a VPN. A VPN is simply a network interconnect. Based on some authentication data, allow a host to connect to a private network. If done properly, there is no service to take advantage of and the daemon can run completely without any capabilities or system access, save of course network traffic. You then add firewalling to ensure only the right hosts get to the right services. You can even make this a single stand-alone VPS host to add granularity and separation of privilege, not to mention reduced attack surface.
Your tool is theoretically simpler in practice than either of those methods (I don't find setting up a VPN to be complex, but others might). But it comes with the same risks as OpenSSH, save that (I imagine?) you could set it up to be network-access-only like a VPN. The difference for a VPN being you wouldn't need to configure multiple instances of an application.
You seem to be under the misapprehension that VPN code never has any bugs in it.
Everything has bugs. If you're trying to make a point about the idea that this software was created with the intent that "it's small so it can't have any bugs", you're right, that probably has bugs too, and relying on that factor alone would be a horrible way to secure a network service. Hence why I suggested a VPN segregated from the rest of the system, or systems on the whole.
Colin, I've always thought of running SSH on some other port as the
equivalent of painting your front door camouflage with the hope
attackers might overlook it -- in other words, pointless. It might stop
some of the "log spam" from dumb brute force attacks on port 22, but if
a port scan was done to find the sshd listening port (i.e. a smart
attacker), then you still get attacked and log spammed. As far as I
know, this issue is best handled by pf.
I'm probably missing something obvious, but I don't understand how spipe/spiped on another port fixes (or helps) the brute force problem?
I'm probably missing something obvious, but I don't understand how spipe/spiped on another port fixes (or helps) the brute force problem?
People can connect to spiped, but nothing will go through to sshd unless they can complete the handshake (which requires an unfeasible amount of work unless the person connecting holds the spiped secret).
Interesting. I'll check it out. Thanks!
I've always wanted something like spamd(8) for ssh, so everyone who normally blocks inbound port 22 can work together to thwart the brute force attacks by providing a honeypot/sandtrap/tarpit for the attack scripts to waste all of their time and resources.
I've always wanted something like spamd(8) for ssh, so everyone who normally blocks inbound port 22 can work together to thwart the brute force attacks by providing a honeypot/sandtrap/tarpit for the attack scripts to waste all of their time and resources.
"in other words, pointless. It might stop some of the "log spam".."
You kind-of contradicted yourself in four words there.. Afaik, it stops the most common "attacks", the automated drive-by password-guessing attacks. Far from pointless.
You kind-of contradicted yourself in four words there.. Afaik, it stops the most common "attacks", the automated drive-by password-guessing attacks. Far from pointless.
I can see your point but you're not looking at the problem realistically;
dumb attack tools get smarter over time. Let's say a dumb attack tool
tries to access the root account over ssh on port 22. If enough people
disable root ssh access or stop running ssh on port 22, then you should
expect the attack tools to evolve (and yes, they have) to trying other
accounts and other ports. The end result is your logs still get filled
with crap. If your goal is to prevent your logs from being filled with
crap, then changing ports won't solve the problem.
Personally, I don't know of a working solution to prevent ssh brute force attacks from filling up our logs with crap, but something like spamd(8) for sshd might make the attackers give up (or at least learn to leave the people running it alone).
http://www.openbsd.org/spamd/
http://www.openbsd.org/cgi-bin/man.cgi?query=spamd&sekti...
http://www.openbsd.org/cgi-bin/man.cgi?query=spamd-setup&...
The goal of spamd(8) is making it painful, slow, and expensive for spammers while keeping mail services running. It seems Colin is headed in a vaguely similar direction with spipe/spiped albeit without the intentionally sadistic tar-pitting and stuttering of spamd(8).
Personally, I don't know of a working solution to prevent ssh brute force attacks from filling up our logs with crap, but something like spamd(8) for sshd might make the attackers give up (or at least learn to leave the people running it alone).
http://www.openbsd.org/spamd/
http://www.openbsd.org/cgi-bin/man.cgi?query=spamd&sekti...
http://www.openbsd.org/cgi-bin/man.cgi?query=spamd-setup&...
The goal of spamd(8) is making it painful, slow, and expensive for spammers while keeping mail services running. It seems Colin is headed in a vaguely similar direction with spipe/spiped albeit without the intentionally sadistic tar-pitting and stuttering of spamd(8).
yup, something like spamd for ssh could be cool..
But evolving to crack other usernames and other ports is not so easy, and there's not even any incentive to do that yet.
"If your goal is to prevent your logs from being filled with crap, then changing ports won't solve the problem"
..Looking at my logs..
Yes, it did actually :)
(edit: typo)
But evolving to crack other usernames and other ports is not so easy, and there's not even any incentive to do that yet.
"If your goal is to prevent your logs from being filled with crap, then changing ports won't solve the problem"
..Looking at my logs..
Yes, it did actually :)
(edit: typo)
I'm glad using an alternate port still works for you. On my home network,
I lost my patience with the attacks a while ago and simply prevented sshd
binding to the external interface of my firewall(s). Since I don't need
external access to my home network, it was the easy way out.
Some friends of mine run tunnelr.com where they run sshd on multiple ports, and they get hammered by attacks on a regular basis. Since they are a VPN service with tons of bandwidth, they're a prime target, and they get attacked on any port where sshd is running. It's a mess, and the approach that's working for you, doesn't work for them (and others I know).
Just out of curiosity, and without naming any (domain) names, would you consider the systems where you're running on a nonstandard port to be "popular" or "high value" or "interesting" targets for attackers? (i.e. subjected to targeted attacks rather than just the typical bulk/blind/dumb attacks)?
Some friends of mine run tunnelr.com where they run sshd on multiple ports, and they get hammered by attacks on a regular basis. Since they are a VPN service with tons of bandwidth, they're a prime target, and they get attacked on any port where sshd is running. It's a mess, and the approach that's working for you, doesn't work for them (and others I know).
Just out of curiosity, and without naming any (domain) names, would you consider the systems where you're running on a nonstandard port to be "popular" or "high value" or "interesting" targets for attackers? (i.e. subjected to targeted attacks rather than just the typical bulk/blind/dumb attacks)?
"Just out of curiosity, and without naming any (domain) names, would you consider the systems where you're running on a nonstandard port to be "popular" or "high value" or "interesting" targets for attackers? (i.e. subjected to targeted attacks rather than just the typical bulk/blind/dumb attacks)?"
No, not at all.
But remember what we are talking about here. You said moving ports is "pointless", and I argued that's it's not because it stops a lot of drive-by attacks. Instead of simply acknowledging this, you spent a lot of words arguing a straw-man.
No, not at all.
But remember what we are talking about here. You said moving ports is "pointless", and I argued that's it's not because it stops a lot of drive-by attacks. Instead of simply acknowledging this, you spent a lot of words arguing a straw-man.
Actually, I didn't intend to be argumentative, contradict you, use
straw man nonsense, or try to convince you of anything. I'm sorry if
it came across that way.
What I'd like to do is just grasp the reason for the discrepancy between your experience and my own (and folks I know)? Obviously, changing ports works for you, and not for me, so I'm trying to figure out why. Maybe it's the type of sites you run, how you host them (racked, cloud, home, ...), upstream/provider filtering, where you're located, or something else?
What I'd like to do is just grasp the reason for the discrepancy between your experience and my own (and folks I know)? Obviously, changing ports works for you, and not for me, so I'm trying to figure out why. Maybe it's the type of sites you run, how you host them (racked, cloud, home, ...), upstream/provider filtering, where you're located, or something else?
My guess is just that my sites fly under the radar because they are unimportant, have little traffic etc..
But if people finds you random (not <1024) ssh port, they probably scanned all of you ports? Which seems drastic. Last time I checked, a few years ago, it was not normal to scan all the ports on a large block of IPs. Maybe that changed, but I have never seen that on the few machines and networks I can check.
Anyway, my point is that targeted attacks is a another ballgame, but that does not make it pointless to protect against the dumb, automated attacks.
But if people finds you random (not <1024) ssh port, they probably scanned all of you ports? Which seems drastic. Last time I checked, a few years ago, it was not normal to scan all the ports on a large block of IPs. Maybe that changed, but I have never seen that on the few machines and networks I can check.
Anyway, my point is that targeted attacks is a another ballgame, but that does not make it pointless to protect against the dumb, automated attacks.
The title made me laugh. Let's protect a very secure, well tested and popular codebase by one that isn't as well tested or popular. Hmmmmm.
Let's protect a large and complicated codebase which has a long history of security vulnerabilities with a very small and easily audited utility.
I totally agree with the principle - smaller code base, considerably less potential for security holes - but in practice, how many eyeballs have audited spiped in its one year since release (and how many of those belonged to crypto-experts - I, for one, couldn't possible spot any vulnerabilities)?
You have a very good track record of course, so that should count in favour of spiped, but on the other hand there's also some value in the massive user base of openSSH, when it comes to auditing.
You have a very good track record of course, so that should count in favour of spiped, but on the other hand there's also some value in the massive user base of openSSH, when it comes to auditing.
What are you protecting? You can't brute force key-based authentication for SSHd.
I'm talking about security vulnerabilities in OpenSSH.
Is there really a history of security vulnerabilities in OpenSSH? (not openssh as patched by idiotic linux distributions)
There have been problems... http://www.openssh.org/security.html
So technically those are all "vulnerabilities in openssh", but most of those are either attacks on the key generation process, X11 forwarding, or privilege escalation on the host machine from a legitimate login - none of which would be protected against by the suggestion in the story.
The first one I see where this defense would actually be helpful is the "September 23, 2003: Portable OpenSSH Multiple PAM vulnerabilities", which only applies against a nonstandard PAM configuration, where enabling PAM at all is something openssh doesn't recommend. The next is "April 21, 2002: Buffer overflow in AFS/Kerberos token passing code" - again, an obscure login protocol that's disabled by default.
The first vulnerability that would affect "normal" users is the "Feb 8, 2001: SSH-1 Daemon CRC32 Compensation Attack Detector Vulnerability", 11 years ago. Even if you want to count the "nonstandard config" vulnerabilities, we're still talking 8 years since any real attack, for possibly the most high-profile encryption software in the world. That's a track record I'd put more faith in than spiped's.
If you want to secure your openssh you'd get a better return on your time (IMO) by disabling any login methods you don't use (or, even better, switching entirely to single-use keys), and rate limiting login attempts from a single IP.
The first one I see where this defense would actually be helpful is the "September 23, 2003: Portable OpenSSH Multiple PAM vulnerabilities", which only applies against a nonstandard PAM configuration, where enabling PAM at all is something openssh doesn't recommend. The next is "April 21, 2002: Buffer overflow in AFS/Kerberos token passing code" - again, an obscure login protocol that's disabled by default.
The first vulnerability that would affect "normal" users is the "Feb 8, 2001: SSH-1 Daemon CRC32 Compensation Attack Detector Vulnerability", 11 years ago. Even if you want to count the "nonstandard config" vulnerabilities, we're still talking 8 years since any real attack, for possibly the most high-profile encryption software in the world. That's a track record I'd put more faith in than spiped's.
If you want to secure your openssh you'd get a better return on your time (IMO) by disabling any login methods you don't use (or, even better, switching entirely to single-use keys), and rate limiting login attempts from a single IP.
I completely agree, but I thought it only fair to list the vulnerabilities. On the other hand, most of it didn't make sense to me... Though that should worry me more than you and thanks for your explanation. :)
Having said that, the vulnerability you pointed out was in SSH1, right? Isn't that depreciated?
Having said that, the vulnerability you pointed out was in SSH1, right? Isn't that depreciated?
>Having said that, the vulnerability you pointed out was in SSH1, right? Isn't that depreciated?
It is now. I'm not sure it was in 2001.
It is now. I'm not sure it was in 2001.
Yes
Bookmarking this comment as an example of one approach to security... we'll see how it turns out!
You can brute force sshd:
while true
select a random common login
select a random common password
loop over many many hosts having sshd
try to connect to sshd
Well that's their fault for using passwords, not keys.
It is the default configuration on ubuntu boxes
It's the default on every install I have ever used, as you have to be able to install your key in the first place. That isn't solved by `spiped`.
Worst case scenario, there's an exploit that punches through spiped. Now you can connect to SSH in the usual manner.
As far as I can see, in the absolute worst case scenario, you end up in an equivalent position to not running spiped at all - it adds, it does not subtract. It's like putting a second locked door in front of the front door to your house - if someone bypasses it, it doesn't make the other door less secure.
As far as I can see, in the absolute worst case scenario, you end up in an equivalent position to not running spiped at all - it adds, it does not subtract. It's like putting a second locked door in front of the front door to your house - if someone bypasses it, it doesn't make the other door less secure.
I agree, but it could give a false sense of security to a novice.
What if they set up `spiped` and password authentication?
Also, once they get to `spiped` they are on your system. If there is a whole to run arbitrary commands from `spiped`, you're stuffed.
What if they set up `spiped` and password authentication?
Also, once they get to `spiped` they are on your system. If there is a whole to run arbitrary commands from `spiped`, you're stuffed.
> If there is a whole to run arbitrary commands from `spiped`, you're stuffed.
Oh, duh, of course. Forget what I said before. Unless you're running spiped in a separate VM or jail.
Oh, duh, of course. Forget what I said before. Unless you're running spiped in a separate VM or jail.
This feature seems to be offering similar functionality to OpenVPN's tls-auth which requires an additional key before a connection is allowed. Another option to get similar functionality would be to use port knocking [1][2] but spiped gives other useful functionality too.
[1] http://www.portknocking.org/ [2] http://www.thoughtcrime.org/software/knockknock/
[1] http://www.portknocking.org/ [2] http://www.thoughtcrime.org/software/knockknock/
I was intending to mention port knocking in that blog post but didn't find a convenient point to fit it in. Yes, it's definitely an alternative solution to the "let me connect to sshd but not anyone else" problem.
This looks incredibly useful, thanks.
I'd been kinda using two methods to do this already
How ready-for-production is the toolset ?
I'd been kinda using two methods to do this already
* OpenVPN plus tun firewalling
* stunnel
How ready-for-production is the toolset ?
Yeah, there's several ways to do this -- 'ssh -L' is probably the most common, followed by stunnel, followed by various types of VPN -- but for the use cases I care about none of them fit very well.
How ready-for-production is the toolset?
I've been using spiped in production for the past year. I've been using spipe for about two days (since I only wrote it two days ago). I'd characterize both as being very production-ready; the amount of code is small enough (about 4000 lines of code, of which 3/4 is library code shared with other projects I've released) that there's very little room for bugs to hide.
How ready-for-production is the toolset?
I've been using spiped in production for the past year. I've been using spipe for about two days (since I only wrote it two days ago). I'd characterize both as being very production-ready; the amount of code is small enough (about 4000 lines of code, of which 3/4 is library code shared with other projects I've released) that there's very little room for bugs to hide.
I realize it's not quite the same thing, but how do you think this compares to Port Knocking?
I ask because I have the impression that using GCM would further simplify/reduce the amount of code and because I'm interested in learning what goes into these decisions. BTW, I'm perfectly happy with an answer of "I can't even explain it because you don't know enough yet." if that's the case.
[EDIT] Extra information for not-yet-crypto-nerds:
The "modes" i'm referring to are cipher block modes. Cipher block modes allow repeated application of a block cipher to long messages (because a block cipher's input must be a fixed length - that's what makes it a "block" cipher rather than a "stream" cipher).
Using a block cipher only gives you confidentiality; it does not verify the inegrity or the source of the message. For those, a message authentication code (MAC) is needed. There are several MAC algorithms.
There are some block cipher modes (including GCM) that provide both authentication and encryption because they include a MAC algorithm.
http://en.wikipedia.org/wiki/Block_cipher_modes_of_operation
http://en.wikipedia.org/wiki/Message_authentication_code