Tokenized Tokens(fly.io)
fly.io
Tokenized Tokens
https://fly.io/blog/tokenized-tokens/
32 comments
Good point about limiting request types.
I created a simple proxy application at my job to handle authenticating to various Atlassian services, like Jira, using OAuth 1.1. I used the proxy to not only perform the OAuth operations on behalf of a client, I also used it to limit operations to read-only. It works great, and was pretty easy to make.
As the article says, creating these proxies allows for a much smaller attack surface area. The proxy itself is relatively simple.
As an aside, the whole reason I needed the proxy I made was because Atlassian uses OAuth 1.1 instead of OAuth 2, or at least they did in the past with their server products. The cloud products may do something different now.
The problem with OAuth 1.1 is that it requires clients to perform cryptographic operations. These tend to be complicated and tricky to implement. OAuth 2 fixed that by eliminating the need for cryptographic operations or pushing them to the server instead of the client.
I created a simple proxy application at my job to handle authenticating to various Atlassian services, like Jira, using OAuth 1.1. I used the proxy to not only perform the OAuth operations on behalf of a client, I also used it to limit operations to read-only. It works great, and was pretty easy to make.
As the article says, creating these proxies allows for a much smaller attack surface area. The proxy itself is relatively simple.
As an aside, the whole reason I needed the proxy I made was because Atlassian uses OAuth 1.1 instead of OAuth 2, or at least they did in the past with their server products. The cloud products may do something different now.
The problem with OAuth 1.1 is that it requires clients to perform cryptographic operations. These tend to be complicated and tricky to implement. OAuth 2 fixed that by eliminating the need for cryptographic operations or pushing them to the server instead of the client.
Shameless plug, but we have a product at Evervault[0] called Outbound Relay[1] that’s basically a hosted, managed version of this flow. Secrets can be encrypted anywhere you collect them (either from third-party APIs or directly from the user in their browser) and then used any time you send a request to a downstream API provider.
[0]: https://evervault.com
[1]: https://docs.evervault.com/products/outbound-relay
[0]: https://evervault.com
[1]: https://docs.evervault.com/products/outbound-relay
This is pretty cool and I think the right direction. Would like to see more companies do this. I've built out systems like this in the past that essentially only handle tokens instead of sensitive data whether it's secrets, PII, etc.
You can also take this a step further and do mathematical operations on encrypted data using homomorphic encryption without ever having to decrypt the data.
Just one small nitpick (mainly because I worked in this space for a few years) is that tokens and encrypted values are different. Tokens aren't encrypted and instead randomly generated using a KV pair look up table so that an attacker could never reverse engineer them. Whereas encrypted values obviously use a key (whether symmetric or asymmetric) and could theoretically (although pretty much never practically if you're using something like AES256) be hacked if someone got the key.
You can also take this a step further and do mathematical operations on encrypted data using homomorphic encryption without ever having to decrypt the data.
Just one small nitpick (mainly because I worked in this space for a few years) is that tokens and encrypted values are different. Tokens aren't encrypted and instead randomly generated using a KV pair look up table so that an attacker could never reverse engineer them. Whereas encrypted values obviously use a key (whether symmetric or asymmetric) and could theoretically (although pretty much never practically if you're using something like AES256) be hacked if someone got the key.
Psyched to get this out (I mean, I didn't do any of the work, but I'm still psyched it's getting out). I'm also a little bit surprised Vault doesn't already have a feature like this as a Secrets Engine.
This is really cool, but I think it ends up just being a complicated decryption oracle?
If I compromise your rails app, and (hypothetically) Stripe allows me to specify the message as it appears on a users credit card statement, could I not just ask it to insert the API key in that field as well and then check my bank statement? This gets easier if there is something where a value gets reflected back to the user, say an SSO error message.
My apology if there is already a protection for this, but I didn't see any obvious use restrictions in the Github README example.
If I compromise your rails app, and (hypothetically) Stripe allows me to specify the message as it appears on a users credit card statement, could I not just ask it to insert the API key in that field as well and then check my bank statement? This gets easier if there is something where a value gets reflected back to the user, say an SSO error message.
My apology if there is already a protection for this, but I didn't see any obvious use restrictions in the Github README example.
As I read the docs, I think the specification for how the secret should be injected lives "statically" within the Tokenizer service, and is not configurable by the internal service making the request.
I.e, if you only get control of the Rails app would need to find an api.stripe.com endpoint that reflects back the authentication header.
---- EDIT: No, I misunderstood it completely, you are right. But hmm. One way I can think of solving what you mentioned is if the token itself contains the processor parameters. That way it wouldn't be possible to change how the templating works after the secret have been tokenised (i.e by an attacker)
I.e, if you only get control of the Rails app would need to find an api.stripe.com endpoint that reflects back the authentication header.
---- EDIT: No, I misunderstood it completely, you are right. But hmm. One way I can think of solving what you mentioned is if the token itself contains the processor parameters. That way it wouldn't be possible to change how the templating works after the secret have been tokenised (i.e by an attacker)
Self-reply since I can't edit my comment anymore. It looks like they are fixing this:
https://github.com/superfly/tokenizer/pull/9
https://github.com/superfly/tokenizer/pull/9
The problem described in the comment above isn't really a problem: you can only inject secrets into headers (like `X-Whatever-API-Key`); you can't inject it into the payload of a request, to, like, ask for an invoice to printed with the API key as an invoice ID.
But there are headers that can get reflected --- too many of them, in too many different environments, so we're just whitelisting which headers you can use on a per-secret basis.
The much, much better vulnerability is the byte-at-a-time attack Jann found. Also fixed! A great bug.
But there are headers that can get reflected --- too many of them, in too many different environments, so we're just whitelisting which headers you can use on a per-secret basis.
The much, much better vulnerability is the byte-at-a-time attack Jann found. Also fixed! A great bug.
Great project and write up!
Did the team consider developing a custom secret engine [1] for Vault? or is it that the specific dances between Rails, tokenizer, ssokenizer cannot be accommodated by a secret engine?
[1]: https://developer.hashicorp.com/vault/tutorials/custom-secre...
Did the team consider developing a custom secret engine [1] for Vault? or is it that the specific dances between Rails, tokenizer, ssokenizer cannot be accommodated by a secret engine?
[1]: https://developer.hashicorp.com/vault/tutorials/custom-secre...
We're really picky about how our existing Vault clusters are exposed to applications, so building a Secret Engine for this would have required us to run an entire new Vault cluster. Moreover, we're moving away from Vault for a bunch of use cases (not all of them! we'll be running Vault indefinitely) --- not because of any failing of Vault, but because at this point we understand our needs very well, and operational legibility has become a really big priority. This also has a clearer path to integrating with our internal Macaroon tokens.
I have to imagine somebody is going to build a Secret Engine that does this.
I have to imagine somebody is going to build a Secret Engine that does this.
Yes, this is cool - thanks to fly.io for documenting and open-sourcing it!
Summary: fly.io wrote two new programs ("some little security thingies") to reduce the attack surface of their codebase.
1. "Tokenizer is an HTTP proxy that injects third party authentication credentials into requests. Clients encrypt third party secrets using the proxy's public key. When the client wants to send a request to the third party service, it does so via the proxy, sending along the encrypted secret in the Proxy-Tokenizer header. The proxy decrypts the secret and injects it into the client's request. To ensure that encrypted secrets can only be used by authorized clients, the encrypted data also includes instructions on authenticating the client."
https://github.com/superfly/tokenizer
2. "Ssokenizer provides a layer of abstraction for applications wanting to authenticate users and access 3rd party APIs via OAuth, but not wanting to directly handle users' API tokens. Ssokenizer is responsible for performing the OAuth dance, obtaining the user's OAuth access token. The token is then encrypted for use with the tokenizer HTTP proxy. By delegating OAuth authentication to ssokenizer and access token usage to tokenizer, applications limit the risk of tokens being lost, stolen, or misused."
https://github.com/superfly/ssokenizer/
If these sound interesting to you, click the submitted link for the "big long essay about how the thingies came to be."
1. "Tokenizer is an HTTP proxy that injects third party authentication credentials into requests. Clients encrypt third party secrets using the proxy's public key. When the client wants to send a request to the third party service, it does so via the proxy, sending along the encrypted secret in the Proxy-Tokenizer header. The proxy decrypts the secret and injects it into the client's request. To ensure that encrypted secrets can only be used by authorized clients, the encrypted data also includes instructions on authenticating the client."
https://github.com/superfly/tokenizer
2. "Ssokenizer provides a layer of abstraction for applications wanting to authenticate users and access 3rd party APIs via OAuth, but not wanting to directly handle users' API tokens. Ssokenizer is responsible for performing the OAuth dance, obtaining the user's OAuth access token. The token is then encrypted for use with the tokenizer HTTP proxy. By delegating OAuth authentication to ssokenizer and access token usage to tokenizer, applications limit the risk of tokens being lost, stolen, or misused."
https://github.com/superfly/ssokenizer/
If these sound interesting to you, click the submitted link for the "big long essay about how the thingies came to be."
To be fair to the author of that blog post, it was a pretty good non top much fluff read.
I wish more things / ecosystems supported mTLS. Then you aren’t storing any long lived tokens. You exchange per instance metadata with key material to generate client tls certs that are then used to auth server calls. The certs have a short lifetime.
The same thing occurred to me while reading this.
I suspect mTLS adoption has been slow because it’s easier to reason about authentication when the mechanics are “closer” to your application code. The mental model of bearer tokens in HTTP headers is pretty easy. Using mTLS requires understanding a lot more moving parts, and TLS still feels like a magical black box in many ways.
Are there any libraries you would recommend that provide a good developer experience around using mTLS?
I suspect mTLS adoption has been slow because it’s easier to reason about authentication when the mechanics are “closer” to your application code. The mental model of bearer tokens in HTTP headers is pretty easy. Using mTLS requires understanding a lot more moving parts, and TLS still feels like a magical black box in many ways.
Are there any libraries you would recommend that provide a good developer experience around using mTLS?
We’ve used Istio for some and a open source system called AthenZ to manage the key material.
Private Key is held in an environment variable called 'OPEN_KEY'. Wouldn't they be better served retrieving the key from Vault or KMS for revocation purposes?
In practice, we do store things in Vault, but we expose that to VMs (where this runs in our architecture) via environment variables injected into the entrypoint process by our init through our orchestrator. You could write direct Vault integration, or do what most people probably do and just use Vault to set an environment variable.
Hey, thanks for clarifying that for me!
I could imagine developers using something like this to get credentials off laptops.
This looks great. A little unfortunate that this effectively requires plaintext http/1.1 (and won't work with "standard" encrypted http/2) - but that's not fly.io's fault.
First read this as “Tokenized Tolkiens”. What a let-down.
South Park did an episode about this
Ahhh just for fun... https://www.youtube.com/watch?v=L9fyB6iJvTU
[deleted]
It’s cool, but the http connection to the proxy makes this only viable with the “vpn” tunnel they use. Otherwise MITM becomes a thing.
Also, you’ve gotten the secret off the client machine, but the attacker can still do anything the secret can do by using the proxy? Perhaps I’m missing something.
Also, you’ve gotten the secret off the client machine, but the attacker can still do anything the secret can do by using the proxy? Perhaps I’m missing something.
We don't need it, nor does anyone who can set up a WireGuard tunnel between their app instance and their proxy instance, or any other IP-level secure channel, but if you really cared you could write the TLS termination logic to make this work --- every MITM testing proxy has it, and there's Go code to do it. It's just pointless in our environment.
The attacker can currently do anything with the secret by interacting with the sites allowlisted for that secret, but they can't exfiltrate the secret, which is the goal of this security control. You can do better, if you like, by further locking down which endpoints they can call, but the wins past "log carefully and no exfiltration" get smaller and smaller, and at some point you're burning time that can be spent more productively on unrelated controls.
If you get what it's doing, you get it. :)
The attacker can currently do anything with the secret by interacting with the sites allowlisted for that secret, but they can't exfiltrate the secret, which is the goal of this security control. You can do better, if you like, by further locking down which endpoints they can call, but the wins past "log carefully and no exfiltration" get smaller and smaller, and at some point you're burning time that can be spent more productively on unrelated controls.
If you get what it's doing, you get it. :)
I think I get what you’re doing, I’m just struggling to see the significance. The OAuth2 token is supposed to only provide the access the client needs. So you don’t get the raw token and instead need to interact via the proxy, so what? What have you blocked the attacker from doing? [edit] I do see intrinsic value in being able to prevent direct access to the tokens and secrets, it just doesn’t look the game changer implied here. Not to me at least.
None of this is a game changer. It's just a way to handle secrets and use them with code driven by Rails without giving Rails access to the secret bits in its memory.
> What have you blocked the attacker from doing?
Not blocked necessarily, but if they want to leverage a stolen token, they’re now forced down a more difficult and highly visible pathway.
You can imagine anomaly detection along the lines if “hey your rails app just made a type of request that it has never made before”, but even just monitoring the metrics of the proxy could tip you off if something is going on.
Not blocked necessarily, but if they want to leverage a stolen token, they’re now forced down a more difficult and highly visible pathway.
You can imagine anomaly detection along the lines if “hey your rails app just made a type of request that it has never made before”, but even just monitoring the metrics of the proxy could tip you off if something is going on.
[deleted]
Another thing you can do with a proxy like this is limit the types of requests which can be made to the 3rd party service. This is very useful if the 3rd party service doesn't support fine-grained permissions for its API keys and you don't want your application to have full access to the 3rd party service.
(I implemented something very similar for my company, which I described here in case anyone is curious: https://sslmate.com/resources/protecting_customer_credential...)