Duplicate SSH Keys Everywhere(blog.shodan.io)
blog.shodan.io
Duplicate SSH Keys Everywhere
https://blog.shodan.io/duplicate-ssh-keys-everywhere/
13 comments
This is not really news, this was already discovered by Nadia Heninger and others in a paper in 2012. From their faq about their research:
"We found significant numbers of insecure RSA and DSA public keys in use on the Internet: we found that 5% of HTTPS hosts and nearly 10% of SSH hosts shared keys for reasons we considered a vulnerability."
See here: https://factorable.net/faq.html
Mostly embedded devices that auto-generate their key on bootup. There was a pretty interesting talk at ccc where part of this research is mentioned:
http://media.ccc.de/browse/congress/2012/29c3-5275-en-factha...
This also applies to those of you building "template" VMs in VMWare or Xen -- before you save that template please delete the SSH keys in /etc/sshd/ so they are generated on first boot of the deployed template!
It would not surprise me if there are other bits of unique info on the template VM that need to be cleared before freezing the image, e.g. hostnames.
This is a solved problem in Windows. The sysprep tool wipes out the hostname, security keys, user accounts and a bunch of other things one can configure, so the image initializes properly the first time it's booted.
I am curious if there are similar tools for Linux & Unix that the image builders simply failed to use here, or if this points to an opportunity to build a useful tool.
This is a solved problem in Windows. The sysprep tool wipes out the hostname, security keys, user accounts and a bunch of other things one can configure, so the image initializes properly the first time it's booted.
I am curious if there are similar tools for Linux & Unix that the image builders simply failed to use here, or if this points to an opportunity to build a useful tool.
There's virt-sysprep [1] and sys-unconfig [2].
[1] http://libguestfs.org/virt-sysprep.1.html
[2] http://linux.die.net/man/8/sys-unconfig
[1] http://libguestfs.org/virt-sysprep.1.html
[2] http://linux.die.net/man/8/sys-unconfig
virt-sysprep from libguestfs does this.
http://libguestfs.org/virt-sysprep.1.html
http://libguestfs.org/virt-sysprep.1.html
in VMware, you use Customization Specifications to do the equivalent to sysprep for Linux (and sysprep on Windows VMs)
With standard ssh-keys, you now have a new problem: how do you know the key of your new vm? Trust-on-first-use is one option -- but a better option would probably be to move to ssh server/user certificates, and generate and sign a new server cert outside the vm, as part of finalizing/readying it for deployment. Might not work well with elastic generation of vms based on load (not sure how well ssh deals with mulitple CAs -- one might imagine a structure with an ultimately trusted, possibly long-lived, off-line cert on the top, and the various sub-CAs organized based on security domains (eg: elastic vm, user auth etc)).
Still doesn't exist any sane, open automation for that, as far as I know. Maybe we should be happy people use keys at all, never mind if the trust/validation is flawed... (not to mention keys never expire, and can only be revoked via a black-list).
Still doesn't exist any sane, open automation for that, as far as I know. Maybe we should be happy people use keys at all, never mind if the trust/validation is flawed... (not to mention keys never expire, and can only be revoked via a black-list).
I've been thinking about adding server key synchronization (optional, of course!) across a Userify server group (cluster) but it seems even riskier than trust-on-first-use. What are your thoughts? Userify is definitely getting expiring keys soon (tracking when the key was last updated).
To be perfectly honest, as I understand what userify does (I'm not a user) -- the whole platform would benefit [ed:from] transition[ing] to ssh certificates[1]. While key synchronization is better than doing... nothing (which a lot of people do, including me, from time to time :-). I think key management for public key authentication is pretty much a solved problem: run your own CA (or delegate to someone, like Userify).
There are still real issues with managing root keys etc -- but with cert support in openssh (client and server) that provides a sane basis for trust management.
For Userify, the service (as opposed to keys, auth, servers and users: the problem) -- I think spending some time on a UI to manage certs/ca for ssh, documenting how to use it for user-keys/certs -- and offering to run a CA per Userify-account (ie: generate/manage the root CA-cert, allowing "instant security" by deploying the Userify-user's CA root cert on servers, and having a service to sign public-keys with that key) could be one very sensible approach.
There might be some added benefit with an "always on"-service, by issuing short-lived user certs (valid for, say 8 hours a la kerberos tickets) -- but for that to be useful, perhaps some work would be needed with forking putty for windows users and/or patching ssh-agent/ssh for nix/OS X users. The basic functionality is there, but the UI/UX is lacking. Note that a user can keep the secret key* for as long as he/she is comfortable -- it's only the signed public cert that needs to be updated (as the previous one expires).
Not sure if that answers your question, but those are some of my thoughts on the matter :-)
[1] http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man1/...
There are still real issues with managing root keys etc -- but with cert support in openssh (client and server) that provides a sane basis for trust management.
For Userify, the service (as opposed to keys, auth, servers and users: the problem) -- I think spending some time on a UI to manage certs/ca for ssh, documenting how to use it for user-keys/certs -- and offering to run a CA per Userify-account (ie: generate/manage the root CA-cert, allowing "instant security" by deploying the Userify-user's CA root cert on servers, and having a service to sign public-keys with that key) could be one very sensible approach.
There might be some added benefit with an "always on"-service, by issuing short-lived user certs (valid for, say 8 hours a la kerberos tickets) -- but for that to be useful, perhaps some work would be needed with forking putty for windows users and/or patching ssh-agent/ssh for nix/OS X users. The basic functionality is there, but the UI/UX is lacking. Note that a user can keep the secret key* for as long as he/she is comfortable -- it's only the signed public cert that needs to be updated (as the previous one expires).
Not sure if that answers your question, but those are some of my thoughts on the matter :-)
[1] http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man1/...
Excellent thoughts!! This is definitely the sort of problem we're trying to solve. Certificates get us part of the way there in terms of solving the authentication problem, and then we can continue to manage authorization separately.. something to think about.
btw, please feel free to become a user ;)
btw, please feel free to become a user ;)
Please note that mixing standard ssh certs and something like userify should be a good mix. From the man-page above, we note that a cert (the signed public key of a user) can include principals -- that is user ids and hostnames (as well as origninatiing IPs) -- and the cert will only be valid for those combinations.
So if a service provides users with "session-tickets" (eg: certs valid for 8 hours, or 5 days etc) -- the servers need not call-back to userify. Authorization (for at least part of use-cases) becomes decentralized. Add an api for extending the validity of a cert (client does a lookup, userify checks policies, and if everything is ok, signs the key for a new validity-period) -- and everything should work very nicely indeed.
For those interested in ssh certs in general, see also:
https://github.com/cloudtools/ssh-ca
As for the userify shim, even though you've answered my question: https://github.com/userify/shim/issues/4 -- you might still want to include a license file, or something along the lines of: http://www.tarsnap.com/legal.html in the readme.
Might also want to have a free/open licensed spec of the api somewhere, so that if someone were to implement a self-hosting, free/open service that could be made api-compatible with userify, there'd be a clear way to do that legally.
So if a service provides users with "session-tickets" (eg: certs valid for 8 hours, or 5 days etc) -- the servers need not call-back to userify. Authorization (for at least part of use-cases) becomes decentralized. Add an api for extending the validity of a cert (client does a lookup, userify checks policies, and if everything is ok, signs the key for a new validity-period) -- and everything should work very nicely indeed.
For those interested in ssh certs in general, see also:
https://github.com/cloudtools/ssh-ca
As for the userify shim, even though you've answered my question: https://github.com/userify/shim/issues/4 -- you might still want to include a license file, or something along the lines of: http://www.tarsnap.com/legal.html in the readme.
Might also want to have a free/open licensed spec of the api somewhere, so that if someone were to implement a self-hosting, free/open service that could be made api-compatible with userify, there'd be a clear way to do that legally.
> generated on first boot of the deployed template!
I've always been uncomfortable about that. How much boot time entropy can a VM have?
I've always been uncomfortable about that. How much boot time entropy can a VM have?
Depends on your OS :-)
I believe Vagrant takes care of that for you.
On the same topic, what are the other pitfalls of re-using OS images?
I can think of:
- Hostname conflicts
- SSH key duplication
- Driver issues if the hardware is not the same
Are there more?
This seems like a compelling argument for bootstrapping new instances with configuration management, rather than trying to re-use OS images.
I can think of:
- Hostname conflicts
- SSH key duplication
- Driver issues if the hardware is not the same
Are there more?
This seems like a compelling argument for bootstrapping new instances with configuration management, rather than trying to re-use OS images.
Speaking of hostname conflicts: an ISP in Vietnam has configured all their devices to have a reverse DNS name of "localhost"
https://www.shodan.io/search?query=hostname%3Alocalhost
https://www.shodan.io/search?query=hostname%3Alocalhost
I've experienced this too - I seem to get a few people joining my gameserver from that ISP and I notice the DNS appeared as 'localhost'. Not ideal and I imagine it could break certain things. I always assumed it was a misconfigured internet cafe or something. 'Nice' to know it's a wider issue
MAC addresses. HP-UX saves them in its startup configs for some reason. A place I worked at used a sample build for its 'gold' build. All builds had the same MAC. Hilarity ensued.
MAC Address duplication on virtual interfaces for VMs started from a template.
No (mainstream) Hypervisor does this and you'd run into networking problems pretty quickly if any of those VMs share a network.
Libvirt has serious issues with this https://www.redhat.com/archives/libvirt-users/2014-November/...
Never seen this in years of using VMs
An alternative to typical configuration management (puppet, ansible, etc) is making your own OS installer. I've adapted the Ubuntu installer to turn a blank machine into a running instance of our app in just a couple of days. The client just has to pop-in the CD or flash drive, answer a couple of questions, and go grab a cup of tea while it installs.
>in just a couple of days
Days?! That is a crazy-long deploy time.
Days?! That is a crazy-long deploy time.
No, I meant it took me a couple of days to make the installer :-)
.. here I thought you were talking about the normal speed of cloudformation to boot up EC2 instances..
You jest, but our previous "cloud" provider (which we used in 2012/13) took more than that to provision a bog standard Linux VPS - and they often got the distro wrong!
Even things like /etc/machine-id
shared inode number
Another implications of this is that if a security flaw is found in this re-used image, it's trvial to write a worm, as it can very easily seek out and find vulnerable hosts.
Ooops.
Ooops.
This is particularly problematic for virtual machines, which are starved of entropy (no sensors, audio device, etc.).
I have VM images begin without ssh host keys and have a service dependency on haveged: http://www.issihosts.com/haveged/ and http://security.stackexchange.com/questions/34523/is-it-appr...
I have VM images begin without ssh host keys and have a service dependency on haveged: http://www.issihosts.com/haveged/ and http://security.stackexchange.com/questions/34523/is-it-appr...
For Qemu 1.3 and later, there is VirtIORand where you can have VMs attach to a HWRNG on the hypervisor, then food that into the kernel CSPRNG. This should be imaged by default, so on first boot, each VM has enough entropy for key generation.
On Ubuntu, pollinate solves this problem: http://blog.dustinkirkland.com/2014/02/random-seeds-in-ubunt...
Not my domain of expertise; can someone explain why this is problematic?
If you can read the private key from just one of these machines then you can impersonate any of them.
That is: MITM SSH connections to these devices without getting any warning.
Of course you first have to get in a position to MITM the person who connects to these devices.
Also if the underlying cryptography isn't forward-secure, anyone with the private key can go back and read your recorded SSH sessions, including any kind of secrets that you typed into them (or files that you downloaded with scp or rsync). I don't know under what conditions SSH uses or doesn't use forward-secure key exchange methods.
I was surprised to learn that you can't man-in-the-middle an SSH connection if the client is using a key for authentication instead of a password:
http://www.gremwell.com/ssh-mitm-public-key-authentication
You'd still be able to impersonate the server, but that's less useful in the general case unless you can emulate the remote machine convincingly long enough to gather useful information.
http://www.gremwell.com/ssh-mitm-public-key-authentication
You'd still be able to impersonate the server, but that's less useful in the general case unless you can emulate the remote machine convincingly long enough to gather useful information.
I'm not sure what you say is true. On it's face, it doesn't make any sense, if you have completely MiTM's the network connection, and can pass along traffic from each part to the other, I don't see how client key auth would prevent MiTM.
Even the thing you link to actually says:
> The algorithm itself does not protect against active MITM attack, but it makes it impossible for MITM attacker to influence the choice of the shared key (and by extension the session ID) by the victims.
Does not protect against active MITM attack. It _does_ keep an attacker from influencing choice of shared key, but an active MITM attack is still in the middle and doesn't need to influence choice of shared key to mount many many kinds of attacks.
Even the thing you link to actually says:
> The algorithm itself does not protect against active MITM attack, but it makes it impossible for MITM attacker to influence the choice of the shared key (and by extension the session ID) by the victims.
Does not protect against active MITM attack. It _does_ keep an attacker from influencing choice of shared key, but an active MITM attack is still in the middle and doesn't need to influence choice of shared key to mount many many kinds of attacks.
The pubkey authentication will fail because the signature is over the session key. If there's a MITM going on then each side will see a different session key. Check out the RFCs 4253 and the auth one (425?)
I'm confused by this. I know that HTTPS is supposed to be fixing this problem too supposedly. But I don't understand how. If an attacker fully replicates a SSH server, and responds to all client messages with the victim correctly, while then relaying their commands to the real SSH server, and acting as a full SSH client, isn't there still an issue?
The attacker just has to authenticate with the correct signature to each, but if it's an attacker, why would it just pass along the victim's data anyway? The whole point of an MITM is that you can modify data before it hits its target.
The attacker just has to authenticate with the correct signature to each, but if it's an attacker, why would it just pass along the victim's data anyway? The whole point of an MITM is that you can modify data before it hits its target.
In the hypothetical scenario the attacker doesn't have the client's private key so it can't authenticate to the server. It can pass along the session key from the server but then it won't be able to read the data.
Ah, I get it now. It's not just the authentication step, it's the fact that the data is encrypted so only the original client can read it? That's right?
[deleted]
[deleted]
From the very first paragraph of that link: "Should ... an attacker manage to steal the private key ... the connection becomes vulnerable to active man-in-the-middle attacks".
Thus, all these machines with duplicate keys are VERY vulnerable to MITM, because anyone with access to one of them has access to the private key.
Thus, all these machines with duplicate keys are VERY vulnerable to MITM, because anyone with access to one of them has access to the private key.
But in the second paragraph:
"there are no tools implementing MITM against an SSH connection authenticated using public-key method ... Being pressed to produce a PoC for this attack, I have attempted to implement it only to discover it is quite impossible and here is why."
"there are no tools implementing MITM against an SSH connection authenticated using public-key method ... Being pressed to produce a PoC for this attack, I have attempted to implement it only to discover it is quite impossible and here is why."
[deleted]
What about "personal" private keys? I find it a hassle to add new keys for every device to each host I ssh into, so I usually just end up copying them from device to device..
All these questions have insightful answers. I'd say "use separate set of private keys for each originating device".
http://superuser.com/q/189355/33303
http://serverfault.com/q/170682/80866
http://security.stackexchange.com/q/40050/13510
http://security.stackexchange.com/q/10963/13510
http://unix.stackexchange.com/q/27661/14305
http://superuser.com/q/189355/33303
http://serverfault.com/q/170682/80866
http://security.stackexchange.com/q/40050/13510
http://security.stackexchange.com/q/10963/13510
http://unix.stackexchange.com/q/27661/14305
Are you talking about SSH host keys? Those are normally generated automatically, and should be unique for each server.
(These are the things you're asked to verify the first time you connect to a new server.)
(These are the things you're asked to verify the first time you connect to a new server.)
Sorry, meant my key, the one I use to connect with and have added the public part of to numerous servers.
There's nothing wrong with adding your public key to many servers, as long as you keep the private key secure -- almost everyone does this. The article is discussing sharing of SSH host keys, which allows impersonation of an SSH server.
That's what Userify is for... don't copy your keys around by hand! :)
SSH agent forwarding should eliminate the need to do that.
I'm not sure how I feel about Shodan. Their statistics are interesting, but apparently the primary use is finding vulnerable systems and matching exploits?
This is the list of most popular searches: https://www.shodan.io/explore
This is the list of most popular searches: https://www.shodan.io/explore
Given the purpose of the key, this doesn't necessarily seem like a security flaw.
If individual owners ssh on to the system, then it is a flaw, but if the primary purpose is to allow system updates, this seems like a perfectly reasonable approach.
This was also the case with Digital Ocean's droplets on Debian for a time (they were re-imaging without re-genning keys.) It's not as uncommon as it sounds.
Userify (cloud ssh key manager) is getting a feature to force re-generation of server-side keys across an entire cluster remotely.
Userify (cloud ssh key manager) is getting a feature to force re-generation of server-side keys across an entire cluster remotely.
I'd suggest that a larger security issue is that the 250,000 hosts are unnecessarily running SSH to the world. A MITM issue doesn't matter if noone's using the connection!
[deleted]
Dumb question, why not have all processors contain a new hardware RNG based on thermal noise or some such thing? That was even VMs can get some stable entropy.
Great point. and Intel's and AMD's black box RNG generators (RDRAND etc) provide entropy... but they're thought to be compromised (which would be nearly impossible to detect). At least other firmware attacks would/should/might be detectable. By definition, RNG's produce unpredictable output -- detecting predictability is much harder than with other potential attacks (versus, for example, the sum of two numbers, which should always give the same result).
So, FreeBSD and Linux both use hardware RNG's as entropy inputs and mix-in other sources of entropy as well (which hopefully mitigates any loss of available entropy and also adds in other believed-good sources of entropy such as timing, network traffic, etc).
Ubuntu is using a new network-attached source of entropy which is itself constantly reseeded with the network traffic used to access it. (There's some inception joke there somewhere..) Of course, you may not be able to rely on the SSL/TLS connection that you're using to access it, so you might be seeding with an attacker's steady stream of 1's...
Of course, getting access to real hardware entropy in a hypervisored or virtualized cloud server/instance is the second part of the problem.
The third part is to make sure that SSH server and client keys are distributed properly. That's what Userify is for, but it really only helps with the client/user keys.. it doesn't help with server keys. (Yet?)
So, FreeBSD and Linux both use hardware RNG's as entropy inputs and mix-in other sources of entropy as well (which hopefully mitigates any loss of available entropy and also adds in other believed-good sources of entropy such as timing, network traffic, etc).
Ubuntu is using a new network-attached source of entropy which is itself constantly reseeded with the network traffic used to access it. (There's some inception joke there somewhere..) Of course, you may not be able to rely on the SSL/TLS connection that you're using to access it, so you might be seeding with an attacker's steady stream of 1's...
Of course, getting access to real hardware entropy in a hypervisored or virtualized cloud server/instance is the second part of the problem.
The third part is to make sure that SSH server and client keys are distributed properly. That's what Userify is for, but it really only helps with the client/user keys.. it doesn't help with server keys. (Yet?)