Reading Uber’s Internal Emails: Bug Bounty report worth $10K(blog.pentestnepal.tech)
blog.pentestnepal.tech
Reading Uber’s Internal Emails: Bug Bounty report worth $10K
http://blog.pentestnepal.tech/post/149985438982/reading-ubers-internal-emails-uber-bug-bounty
14 comments
At the time of the exploit, there was only one check, the MX record. Also once I claimed the domain, another user cannot claim it from their account. So this could be exploited silently as well because the victimized company won't know about it for a while.
The problem is on SendGrid's side, however I doubt they would have paid $10K for it. He was smart to take it to Uber, who is likely (by far) SendGrid's largest customer, and who certainly has much deeper pockets than SendGrid.
[deleted]
Sendgrid allowed attackers to social engineer control of my company's account and intercept password resets, despite an explicit warning from us a week prior (we received a chat transcript of the failed attempt and let them know that it was not us and someone was actively trying to social engineer access to our account).
Then they had the gall to try to convince me on the phone that it must have been my fault (after our blog post about it blew up).
Needless to say, I think they are terrible.
Previous HN discussion: https://news.ycombinator.com/item?id=7476836
Then they had the gall to try to convince me on the phone that it must have been my fault (after our blog post about it blew up).
Needless to say, I think they are terrible.
Previous HN discussion: https://news.ycombinator.com/item?id=7476836
As a former customer (it has admittedly been a few years) I'm not surprised. Sendgrid's entire business is based on price. Emails are one of the few costs that don't really scale that well. They cost a lot more than people think they should. When I list did a cost analysis they beat out a lot of other providers.
I haven't used them in years, so I'm not sure if they are still the low cost leader, but they are definitely the kind of thing where you get what you pay for. Their lack of investment in product and infrastructure showed for the years I used them, and their slowness in delivering updates was incredibly frustrating.
I haven't used them in years, so I'm not sure if they are still the low cost leader, but they are definitely the kind of thing where you get what you pay for. Their lack of investment in product and infrastructure showed for the years I used them, and their slowness in delivering updates was incredibly frustrating.
Who would you recommend then?
I'm not the person you asked but I too have had bad experiences with SendGrid.
It really depends on which aspect of SendGrid we're talking about.
Transactional e-mail? Mailgun is a easy to use API on top of Amazon's SES.
You can even set up incoming e-mail hooks e.g. "When a new e-mail arrives, POST the contents to this address and attach any attachments on the e-mail as file uploads."
Newsletters? Drip campaigns? I have less experience on this side of the realm, but HubSpot has been the best experience I've seen. Their web management UI is also powered by their own open API, if I remember correctly.
It really depends on which aspect of SendGrid we're talking about.
Transactional e-mail? Mailgun is a easy to use API on top of Amazon's SES.
You can even set up incoming e-mail hooks e.g. "When a new e-mail arrives, POST the contents to this address and attach any attachments on the e-mail as file uploads."
Newsletters? Drip campaigns? I have less experience on this side of the realm, but HubSpot has been the best experience I've seen. Their web management UI is also powered by their own open API, if I remember correctly.
I like mailgun, but I am having a devil of a time with them sending to MSN and aol addresses. Basically 0 deliverability. And their tech support is going on 4 days without looking at my ticket. But maybe it's not their fault.
I've had good experiences with Postmark, though I haven't tried them at large scale.
I would recommend them too, I had a bad experience with sensgrid. Went from mandril to sendgrid to postmark and couldn't be happier.
Not Marketo
I was on the fence about leaving SG, now I'm for sure out, thanks.
I think this was submitted because it was discovered on /r/programming
The last time there was a SendGrid article on here, the feedback from the community was far from kind [0]. I again re-iterate that SendGrid has no business sending emails [1].
[0]: https://news.ycombinator.com/item?id=12142728 [1]: https://news.ycombinator.com/item?id=12145019
[0]: https://news.ycombinator.com/item?id=12142728 [1]: https://news.ycombinator.com/item?id=12145019
This! I actually started using them when they helped me out a hackathon but switched to a more dev friendly service.
Ouch. No domain verification required by Sendgrid before allowing you to inject a hook that dumps email contents.
That's much broader than just Uber.
Edit: Yes, it's been fixed, but the fact that it existed for quite some time is still troubling. I'm also curious if the fix retroactively disabled any existing unverified hooks.
That's much broader than just Uber.
Edit: Yes, it's been fixed, but the fact that it existed for quite some time is still troubling. I'm also curious if the fix retroactively disabled any existing unverified hooks.
I believe they did retroactively search for accounts.
Source: I had a number of accounts banned when testing different iterations of this bug.
Source: I had a number of accounts banned when testing different iterations of this bug.
Totally just curious. The law for "exceeds authorized access" is 20 years in prison. I think uber has a bug fixing program. Maybe sendgrid does? Do the DNS carriers? Does his ISP? I read the law. If one of these companies refutes access - this guy is facing 20 years [1]? W(why)TF do people do this? The US government is notoriously creative in these prosecutions [3]. Companies refute access all the time to not look like idiots EVEN WITH a bounty program. [2]. If I came across this I wouldn't be blogging about it for 10k?
[1] "(2) intentionally accesses a computer without authorization or exceeds authorized access, and thereby obtains—(C) information from any protected computer;" (C) except as provided in subparagraphs (E) and (F), a fine under this title, imprisonment for not more than 20 years, or both, in the case of—"
https://en.wikipedia.org/wiki/Computer_Fraud_and_Abuse_Act
[2] http://motherboard.vice.com/blog/facebook-is-refusing-to-pay...
[3] https://en.wikipedia.org/wiki/Aaron_Swartz
[1] "(2) intentionally accesses a computer without authorization or exceeds authorized access, and thereby obtains—(C) information from any protected computer;" (C) except as provided in subparagraphs (E) and (F), a fine under this title, imprisonment for not more than 20 years, or both, in the case of—"
https://en.wikipedia.org/wiki/Computer_Fraud_and_Abuse_Act
[2] http://motherboard.vice.com/blog/facebook-is-refusing-to-pay...
[3] https://en.wikipedia.org/wiki/Aaron_Swartz
Uber's bug bounty program seems like explicit permission to me. The twist here, as you mention, is that it is sendgrid's infrastructure. Chasing bug bounties does seem a bit risky in this "cloud" era where it's not just one entity running the targeted service.
It is not listed as an in-scope domain on Uber's bug bounty policy. You would have to argue that the access was unintentional - that you believed you had authorization to access the server from Uber and that Uber had the ability to assign that authorization on behalf of SendGrid.
That is a bit tricky. You could argue that the MX records for a domain that is "in scope" (www.uber.com) are in play. They don't specifically say you can only follow CNAME and A records...and you do have to follow CNAME for some of the listed domains.
Unfortunately, the law simply does not care about technical details. If you were being prosecuted for this sort of thing and you brought and expert witness or testified yourself about a technical justification such as this, you would be accused of attempting to bamboozle the court with fancy technical wording and it would not go well for you. You might even get found to be in contempt of court. Even the very simple fact that 1% of DNA tests have been shown to be false due to sample contamination and other factors is not permitted to be mentioned in court when DNA matches are being used to prosecute someone as this is too technical. Cases which involve computers fare even worse.
I'm sure a lawyer would find some non technical analogy to help explain the concept.
It isn't a trick...Some of the approved domains have no IP addresses directly associated with them. You have to traverse a reference to another domain name, not on the list, to get to an actual IP address.
Now if that traversal is by something called a CNAME or something called an MX, does it matter?
It isn't a trick...Some of the approved domains have no IP addresses directly associated with them. You have to traverse a reference to another domain name, not on the list, to get to an actual IP address.
Now if that traversal is by something called a CNAME or something called an MX, does it matter?
The scope of Uber's bug bounty program was updated after this report was submitted so that does not count.
Author appears to be in Nepal, which has no extradition treaty with the USA.
Someone reported this same vulnerability to us via HackerOne months ago. We worked with Sendgrid support to re-claim the domain and they said they were urgently working to fix the issue, or not.
Edit: just saw this post was from September. Author probably made thousands in rewards circulating this vulnerability.
Edit: just saw this post was from September. Author probably made thousands in rewards circulating this vulnerability.
I do not believe the author circulated this report to multiple companies, however once it was made public a number of other reporters in the community did and continued to iterate on it until SendGrid fixed the issues.
Source: I am a member of said community: https://bugcrowd.com/bored-engineer, https://hackerone.com/bored-engineer, etc
Source: I am a member of said community: https://bugcrowd.com/bored-engineer, https://hackerone.com/bored-engineer, etc
This is a lot like a bug I found in Heroku's system a few years ago. Basically, if someone doesn't claim the wildcard subdomain for their primary domain and has a wildcard SSL cert anyone could (can?) claim subdomains. A quick google search yielded hundreds of exploitable domains. At the time it seemed like a pretty big vector for phishing.
I have no idea if they fixed this and they gave me a t-shirt.
I have no idea if they fixed this and they gave me a t-shirt.
I can't recall all the exact details, but there is some validation logic in place along the lines of "if there's a wildcard domain installed then newly added subdomains must be on the same account as the wildcard's owner". You could give it a shot, but I don't think this attack would work.
(I used to help maintain the system responsible for this, but don't work there anymore.)
(I used to help maintain the system responsible for this, but don't work there anymore.)
This was probably 2012 or 2013. I'll try to find the write up I sent them, but this check definitely did not exist then
edit: here's what we sent to heroku https://gist.github.com/bdittmer/6461b7a5093acd7d6263
edit: here's what we sent to heroku https://gist.github.com/bdittmer/6461b7a5093acd7d6263
> they gave me a t-shirt.
I had same experience with Salesforce
I had same experience with Salesforce
This looks like a massive security flaw on sendgrid's side. They should use DNS validation like everybody else to prove ownership of the subdomain.
From the article:
>Also at the moment of writing this bug, it has come to my attention that SendGrid has added extra verification which forces you to have a verified domain before adding a inbound parse webhook.
>Also at the moment of writing this bug, it has come to my attention that SendGrid has added extra verification which forces you to have a verified domain before adding a inbound parse webhook.
Technically they did employ some DNS validation. You had to setup a MX record to point to SendGrid before you could add the domain to your account. The problem was in order to send emails from a domain you had to add the same MX record. If you never setup the receiving end as well (on SendGrid) you were vulnerable to a takeover from another SendGrid account.
Hi thank you for the post. If anyone has question, feel free to reach out to me here or at [email protected]. As a writer of this blog, I will be able to provide the feedback necessary and clear any misunderstanding/false positives regarding this.
Looks like they found a similar "3rd party email service / subdomain MX record" exploit with Slack.[1] Although not so severe in Slack's case because it's a lesser used feature. Looks like Slack is using Mailgun.[2]
[1] http://blog.pentestnepal.tech/post/150381068912/how-i-snoope...
[2] http://mxtoolbox.com/SuperTool.aspx?action=mx%3aslack.com&ru...
[1] http://blog.pentestnepal.tech/post/150381068912/how-i-snoope...
[2] http://mxtoolbox.com/SuperTool.aspx?action=mx%3aslack.com&ru...
Honest question: Do bug bounties with such low bounties really do more good than harm?
A bug bounty incentivizes people to look for bugs. But when you find an interesting bug like this you have the option of either having the possibility of making millions from the social engineering possibilities alone, or claiming the bug bounty and get $10k. Of course claiming the bounty is the moral thing to do, but some people with both the skill and motivation to do bug bounties are bound to have lower moral standards.
A bug bounty incentivizes people to look for bugs. But when you find an interesting bug like this you have the option of either having the possibility of making millions from the social engineering possibilities alone, or claiming the bug bounty and get $10k. Of course claiming the bounty is the moral thing to do, but some people with both the skill and motivation to do bug bounties are bound to have lower moral standards.
It's a good question, but I think the answer is no [err, re-reading your question I mean yes... thought it asked whether they did more harm than good]. Consider that you have a population of developers. Some of them are willing to break the law and cause highly probable harm (if only monetary) to someone else. Some are not. Those who are willing are motivated exactly as much whether there is a bug bounty program or not. Those who are not willing are motivated some amount (whether small or large, it is nonzero) by the existence of a bug bounty program to have a look and to turn over the information rather than sell or use it.
So, on the balance, you are increasing motivation for people who are not willing to harm for cash while leaving the motivation of the willing unchanged. I would be tempted to argue that it is flat out unwise to NOT run a bug bounty program, although it would be much smarter to offer larger bounties. I could make an argument for bounties exceeding the projected amount the vulnerability would be worth on the black market, I think, but that's a different subject.
So, on the balance, you are increasing motivation for people who are not willing to harm for cash while leaving the motivation of the willing unchanged. I would be tempted to argue that it is flat out unwise to NOT run a bug bounty program, although it would be much smarter to offer larger bounties. I could make an argument for bounties exceeding the projected amount the vulnerability would be worth on the black market, I think, but that's a different subject.
I think it's hard to monetize most bug bounty bugs.
There isn't really a market for most XSS, CSRF or even RCEs bugs for web properties. Getting the bounty payout for a bug from the owner is often the best deal available. I think the only exception to the no-market situation is browser RCEs and smart phone OS jailbreaks.
There isn't really a market for most XSS, CSRF or even RCEs bugs for web properties. Getting the bounty payout for a bug from the owner is often the best deal available. I think the only exception to the no-market situation is browser RCEs and smart phone OS jailbreaks.
Right, and this one could have easily been monetized because it allows attackers to intercept and read any one of SendGrid's customer's private internal emails!
I don't agree that this could have been easily monetized.
This vulnerability only allows you to intercept future incoming emails that are delivered to sendgrid domains. Even if it did allow access to all of their customers internal emails there aren't many buyers (and no obvious marketplaces) out there for that kind of access.
The risk is simply too high, if you're Lyft (for example) being caught with that access (even if it was never used) would be a possible terminal event for the company. Purchasers would be buying a giant liability and a small competitive advantage.
Maybe there are shady companies or criminal groups who would be interested in the access but even then I feel $10k would be roughly how much money they'd be willing to spend (considering the same amount of money could also buy thousands of verified credit cards or a million email addresses at legitimate providers).
This vulnerability only allows you to intercept future incoming emails that are delivered to sendgrid domains. Even if it did allow access to all of their customers internal emails there aren't many buyers (and no obvious marketplaces) out there for that kind of access.
The risk is simply too high, if you're Lyft (for example) being caught with that access (even if it was never used) would be a possible terminal event for the company. Purchasers would be buying a giant liability and a small competitive advantage.
Maybe there are shady companies or criminal groups who would be interested in the access but even then I feel $10k would be roughly how much money they'd be willing to spend (considering the same amount of money could also buy thousands of verified credit cards or a million email addresses at legitimate providers).
Corporate clients aren't the ones purchasing black market exploits, the marketplaces aren't obvious. Just confirmed with a friend who does not have ties (disclaimer) that this one would be worth ~100k.
Maybe things have changed in recent years but here's some information on what I considered benchmark prices when I was more involved in this kind of thing. I have some background in vulnerability selling and I used to do this for a living (first government, then later tech company bug bounty programs).
The "grey market" prices I've seen (~5 years ago now) from independent researchers, small firms specializing in vulndev for law enforcement / intelligence and the bigger famous security firms go something like this ("black market" criminal markets are similar I hear but I have no first hand knowledge there):
250k would get you a iOS jailbreak (for a recent iOS version)
100k would get you an IE/Firefox/Chrome zero day (though not with a sanbox escape, you'd have to pay separately for a privesc)
~25k for a Linux privesc
For an RCEs in medium popularity services (databases, ftp servers, etc) you'd be looking at something like 10k (or often thrown in free with a bigger purchase or support contract as a good will gesture).
The prices would also change based on the quality of the exploit, how recent the version the exploit targeted was, the amount of exclusivity you want (more for full exclusivity) and whether the research was specially commissioned (if you had asked a firm to look into a specific piece of software the price would be higher). The larger providers also encouraged organizations to sign subscription deals and longer term service contracts that could include free exploits or reduced costs.
That said, there's a lot of information asymmetry in the market so I'd expect a lot of variance in prices.
The "grey market" prices I've seen (~5 years ago now) from independent researchers, small firms specializing in vulndev for law enforcement / intelligence and the bigger famous security firms go something like this ("black market" criminal markets are similar I hear but I have no first hand knowledge there):
250k would get you a iOS jailbreak (for a recent iOS version)
100k would get you an IE/Firefox/Chrome zero day (though not with a sanbox escape, you'd have to pay separately for a privesc)
~25k for a Linux privesc
For an RCEs in medium popularity services (databases, ftp servers, etc) you'd be looking at something like 10k (or often thrown in free with a bigger purchase or support contract as a good will gesture).
The prices would also change based on the quality of the exploit, how recent the version the exploit targeted was, the amount of exclusivity you want (more for full exclusivity) and whether the research was specially commissioned (if you had asked a firm to look into a specific piece of software the price would be higher). The larger providers also encouraged organizations to sign subscription deals and longer term service contracts that could include free exploits or reduced costs.
That said, there's a lot of information asymmetry in the market so I'd expect a lot of variance in prices.
Would have been worth more to SendGrid, don't you think?
Several months ago I reported a bug to a popular service which was allowing password reset & other emails to be retrieved with just a serial id from a service like sendgrid.
It's just me, or $10k is far from being generous?
The bit that's not generous to me is that Sendgrid, who made the mistake, is paying $0.
Why does this comment appear on every bug bounty HN thread? Straight from the horse's mouth [0]:
The black market is very unlikely to be a place you could sell a bug in a specific
website or service. It is not “worth millions”. Please stop repeating this.
What a cute strawman (and completely incorrect strawman from the wrong horse at that, this is Trumpist drivel from the guy who runs the actual bug bounty who of course has some acute rationalizations for underpaying for some of the most intricate technical work in the industry).
Nobody said it was "worth millions" but I have second-degree connections in Scandinavia that would pay 10x, like I said ($100,000).
@collingreene doesn't sound too familiar with these rather-illicit organizations, he strikes me as a product manager type person with a loud voice, not someone who actually has found and sold zero-days before. Maybe he doesn't have the technical acumen to do so, but hey, I'm not one to judge.
It's hard to establish proof that the market value on the black market is, in fact, much higher given that it is the black market (you're not going to find these people on Medium); However, one public example of this is the leaked Stuxnet details showing similarly high 5-digit prices for zero-days.
This isn't a specific bug either (it wasn't "oh, the log files for that one UberEATS micro-service were visible"), this flaw allows you to intercept the emails of pretty much any single one of SendGrid's clients. Imagine the damage someone could do with that, had it gotten into the wrong hands. Only $10k, what an insult.
EDIT: Upon further examination, it turns out that said author also contradicts himself and corroborates my own argument:
https://medium.com/@collingreene/why-product-security-is-har...
Primary source: https://www.wired.com/2016/09/top-shelf-iphone-hack-now-goes...
Nobody said it was "worth millions" but I have second-degree connections in Scandinavia that would pay 10x, like I said ($100,000).
@collingreene doesn't sound too familiar with these rather-illicit organizations, he strikes me as a product manager type person with a loud voice, not someone who actually has found and sold zero-days before. Maybe he doesn't have the technical acumen to do so, but hey, I'm not one to judge.
It's hard to establish proof that the market value on the black market is, in fact, much higher given that it is the black market (you're not going to find these people on Medium); However, one public example of this is the leaked Stuxnet details showing similarly high 5-digit prices for zero-days.
This isn't a specific bug either (it wasn't "oh, the log files for that one UberEATS micro-service were visible"), this flaw allows you to intercept the emails of pretty much any single one of SendGrid's clients. Imagine the damage someone could do with that, had it gotten into the wrong hands. Only $10k, what an insult.
EDIT: Upon further examination, it turns out that said author also contradicts himself and corroborates my own argument:
https://medium.com/@collingreene/why-product-security-is-har...
Primary source: https://www.wired.com/2016/09/top-shelf-iphone-hack-now-goes...
Wow, to further add injury to the insult, United pays bounty hunters with miles:
https://www.united.com/web/en-US/content/Contact/bugbounty.a...
Don't these companies get that researchers need to live?!
https://www.united.com/web/en-US/content/Contact/bugbounty.a...
Don't these companies get that researchers need to live?!
I thought so, too. Uber pays its full time employees some of the highest salaries in the Valley. This exploit could have easily fetched 10x the money on the black market.
The vulnerability is that any SendGrid user could configure a webhook callback which would POST back all received emails for any domain which had its MX set to 'mx.sendgrid.net'. OP exploited this against Uber to receive copies of their emails.
Presumably there was no way to tell from one account that another account is web-hooking your email out from under you. So you have to wonder, if it's as easy as just typing the domain you want to listen in on.... who else was getting all their emails tapped this way?
From Sendgrid's documentation;
Shocking omission by Sendgrid, where's their write-up and apology?