Hologram: taking EC2 Instance Roles everywhere(tech.adroll.com)
tech.adroll.com
Hologram: taking EC2 Instance Roles everywhere
http://tech.adroll.com/blog/ops/2014/12/22/announcing-hologram.html
26 comments
For working with AWS apis, securing access keys, and granting least privilege, the first step to take is to make completely separate AWS accounts per environment. Make a Bigco-Prod, Bigco-Test, Bigco-Dev AWS accounts. Developers only access the Development account, so even if they connect with a highly privileged key / role and a bug wipes out your services, only the Dev account is effected and production keeps on humming. AWS makes this extremely easy to do, they offer consolidated billing to pay one bill for multiple accounts. Takes no more than 30 min to set this all up and I would recommend this for even a company with just 1 developer.
The difficult part of separate AWS accounts is that connecting between them is tricky (Security Groups, VPC, etc.). Not impossible, but a pain to get right. The spectrum between dev and prod isn't so clear-cut for a business with petabytes of data: you often want to develop systems against production data (read-only, of course), and having two copies of the data can be prohibitively expensive. Of course, with the appropriate access policies you can share the data safely, but then what about that IRC server that was non-production but then wound up part of the deployment flow and became mission-critical?
I'm still not sure why it has taken so long for IAM to support this kind of configuration, using VPC as the isolation point.
Our use case is running packer[0], which requires a large IAM profile[1], without needing to create a separate account for AMI creation. It would be awesome to issue credentials that limited Security Group creation/deletion, Instance Creation/Termination, etc to a single, VPC.
[0]: https://www.packer.io/ [1]: https://www.packer.io/docs/builders/amazon.html
Our use case is running packer[0], which requires a large IAM profile[1], without needing to create a separate account for AMI creation. It would be awesome to issue credentials that limited Security Group creation/deletion, Instance Creation/Termination, etc to a single, VPC.
[0]: https://www.packer.io/ [1]: https://www.packer.io/docs/builders/amazon.html
Why not just build using packer's chroot support? No instance creation or security groups and much faster.
I wish AWS made it possible to move reserved instances between accounts. We have reserved instances for our test environment that are in the same account as our production instances, and I'd need to sell them from one account at a loss and repurchase them in the other account.
That is definitely a very good practice.
Still it does not eliminate the need for something like Hologram, since you still need to take care of your keys (both for security reasons) and to make sure your dev env is as close as possible to your production and ...
Didn't mean to imply no need for Hologram or not to keep dev as close as possible to production. However, I do think you should be doing this before looking into a solution like hologram. I've seen many startups that run all their environments on a single account and eventually something bad happens knocking production offline. If you were to use hologram, but still only a single AWS account you could still just as easily kill production.
Absolutely, I've seen similar things. Even after explaining the pitfalls some people are reluctant to do it. Partly because people don't want to embrace the aws ("cloud", the new way of doing things) completely, they still want to think of everything as if it is their own machines in their own data server only amazon plays the role of the ops guys.
I like the angle of how this reduces the diff between development and production. I also agree that AWS keys as managed at many companies are a security risk. However, there's an alternative to Hologram that is built-in to IAM and might provide even more security: multi-factor authentication and the Security Token Service (STS) [0].
MFA requires you to authenticate using both your AWS keys and your MFA device. In other words, an attacker who gains access to your laptop won't gain access to AWS. They'd have to get access to your phone or keyfob too.
When writing your IAM policies, you can require that users authenticate with MFA before they can perfom an action. For example, with a policy of
You shouldn't have to modify existing programs to work with STS. Instead, take out any explicit authorization code, just like you would do to use IAM roles. Then, wrap invocations of the programs with a tool that talks to STS and configures your environment. I've written one in ruby [1], and it should be easy to reimplement if you prefer another language.
Hopefully Hologram can add support for multi-factor authentication in a future release. Right now, it looks like stealing someone's ssh key is all it would take to get access to AWS.
[0] http://docs.aws.amazon.com/IAM/latest/UserGuide/Using_Managi...
[1] https://github.com/lonelyplanet/aws-mfa
MFA requires you to authenticate using both your AWS keys and your MFA device. In other words, an attacker who gains access to your laptop won't gain access to AWS. They'd have to get access to your phone or keyfob too.
When writing your IAM policies, you can require that users authenticate with MFA before they can perfom an action. For example, with a policy of
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": "ec2:Describe*",
"Resource": "*"
}]
}
"Condition": {"Null": {"aws:MultiFactorAuthAge": false}}
You shouldn't have to modify existing programs to work with STS. Instead, take out any explicit authorization code, just like you would do to use IAM roles. Then, wrap invocations of the programs with a tool that talks to STS and configures your environment. I've written one in ruby [1], and it should be easy to reimplement if you prefer another language.
Hopefully Hologram can add support for multi-factor authentication in a future release. Right now, it looks like stealing someone's ssh key is all it would take to get access to AWS.
[0] http://docs.aws.amazon.com/IAM/latest/UserGuide/Using_Managi...
[1] https://github.com/lonelyplanet/aws-mfa
Interesting. Not sure about the requirement for "an LDAP server with sshPublicKey attributes on user records to store their SSH public keys." https://github.com/AdRoll/hologram
I know you can put anything into LDAP, but why treat it like a metadata service? Wouldn't it make more sense to store keys in something like etcd so they're accessible via simple REST api calls? In my experience, the less LDAP does, the happier everyone seems to be.
I know you can put anything into LDAP, but why treat it like a metadata service? Wouldn't it make more sense to store keys in something like etcd so they're accessible via simple REST api calls? In my experience, the less LDAP does, the happier everyone seems to be.
Yeah, we connected with LDAP because that's how AdRoll operates, but we'd like to support a more universal backend eventually. That support just didn't make the cut for open-source release. With a bit of code, you could easily plug in a .txt file or a SQLite database, or even etcd if you really don't want to move a file around.
Many organizations that have the problem this solves are already complex enough to benefit from LDAP. etcd is not appropriate to use outside a controlled server environment because it's not authenticated.
I recently heard of someone using the GitHub API to only allow certain GitHub accounts to have access to a bastion server. Given the target audience, if your company uses GitHub (or perhaps even the internally hosted GitHub Enterprise), that might be a much easier sell than modifying LDAP.
The GitHub API to list a user's keys doesn't require OAuth or any kind of hoop-jumping: https://api.github.com/users/mdaniel/keys
The GitHub API to list a user's keys doesn't require OAuth or any kind of hoop-jumping: https://api.github.com/users/mdaniel/keys
I was also wondering how popular LDAP is? If you are thinking of using Hologram, or can benefit from it, would you please comment and mention if you already use LDAP or some other alternative?
Definitely don't set up LDAP for hologram... but if you already have LDAP set up, it could be worthwhile to modify your setup to support hologram.
But then again, if your org is large enough to have LDAP set up, getting LDAP policies changed to support something like hologram would generally be a non-starter in my experience.
In large enterprises, the LDAP service is a core security service that is rarely if ever modified except for periodic maintenance and upgrades.
Messing up the LDAP service for any reason will compromise or disable all authentication company-wide. It's not fun when the CEO comes over and says that his wifi and email logins aren't working anymore...
But then again, if your org is large enough to have LDAP set up, getting LDAP policies changed to support something like hologram would generally be a non-starter in my experience.
In large enterprises, the LDAP service is a core security service that is rarely if ever modified except for periodic maintenance and upgrades.
Messing up the LDAP service for any reason will compromise or disable all authentication company-wide. It's not fun when the CEO comes over and says that his wifi and email logins aren't working anymore...
I built something very similar to this using iptables and a simple flask app.
Not even close to full-featured, but anyway, here's a link for those who are interested: https://github.com/balanced-ops/docker-host/tree/master/iamp...
I think there's a large gap still to be filled with IAM Roles. Hope to see AWS invest more resources into this.
Not even close to full-featured, but anyway, here's a link for those who are interested: https://github.com/balanced-ops/docker-host/tree/master/iamp...
I think there's a large gap still to be filled with IAM Roles. Hope to see AWS invest more resources into this.
Great news, and very good work, thanks for sharing.
One thing that I am concerned about (or have questions about), is how would you make sure you are not exposing the service to outside, so that when the laptop is on a coffee shop network others can query the credentials?
I mean, it often happens that a developer needs to expose a webserver they are running to outside to showcase something or demo or .... It is not very hard to make a mistake and expose everything right?
Hmm, judging from the source code the default listenAddress is on '0.0.0.0:3100' so it's listening on every interface, but it's instantiated to use TLS and in server.go HandleServerRequest does SSH authentication.
This service is separate from whatever the developer is working on that needs to call to it. If the work in progress exposes temporary credentials it's not the end of the world.
This service is separate from whatever the developer is working on that needs to call to it. If the work in progress exposes temporary credentials it's not the end of the world.
The hologram server is privileged and should run behind a firewall. It will only hand out keys to successfully authenticated (using an SSH challenge-response) clients, so it could be publicly accessible, but why go to that risk?
The hologram agent that runs on your laptop, and could potentially be exposed in a cafe, attaches to lo0, so there's no way somebody on your coffee shop wifi could get the creds.
The hologram agent that runs on your laptop, and could potentially be exposed in a cafe, attaches to lo0, so there's no way somebody on your coffee shop wifi could get the creds.
On the Ec2 side security groups can be used to only allow access for the company IP. Then VPN can be used to secure connections from coffee houses back the the office.
That's what we do at AdRoll, and it seems to work pretty well for us. :)
I love this! We use LDAP in exactly this fashion so this looks like it may be more-or-less a drop-in addition to how we work. Thanks for such a cool project!
I'm really surprised Amazon hasn't developed something like this already. One of my biggest pain points is creating new keys for every project to use locally and then making sure I remember to delete them when I'm done. Not to mention having to recreate them if I do future updates. Really good service here, though, I'll have to check it out!
Yeah, when we spoke with Amazon, they already had a couple of labels for what hologram is, so it sounds like we at AdRoll are not the only AWS customer who has had to roll it ourselves. Hopefully we'll be the last!
This looks very interesting. Any comments on production-worthiness? This isn't your target scenario, but could it evolve into a component for using IAM-style credential management for companies running their own servers instead of using AWS? I'm always interested in new techniques for removing secret keys from server configs.
We've been running Hologram in production at AdRoll for about 2 months now with no issues, and we've eliminated permanent key usage entirely.
We've had a couple of requests internally for a server-oriented agent, and it's something we want to investigate eventually. I'll make a Github issue for it and at least we can have a conversation.
We've had a couple of requests internally for a server-oriented agent, and it's something we want to investigate eventually. I'll make a Github issue for it and at least we can have a conversation.