Sendgrid under siege from hacked accounts(krebsonsecurity.com)
krebsonsecurity.com
Sendgrid under siege from hacked accounts
https://krebsonsecurity.com/2020/08/sendgrid-under-siege-from-hacked-accounts/
32 comments
Same -- I got several e-mails with subject "The following services failed to auto-renew and are about to expire" that I almost believed were real because they came from an actual sendgrid server (wrqvpkzw.outbound-mail.sendgrid.net [149.72.49.233]).
If SendGrid can't develop simple filters to deter phishing e-mails sent through their platform targeting their own service, then obviously they're going to end up with a lot of users being hacked.
I mean seriously, why would an e-mail provider like this allow e-mails to be sent from "[email protected]" and "[email protected]"??
If SendGrid can't develop simple filters to deter phishing e-mails sent through their platform targeting their own service, then obviously they're going to end up with a lot of users being hacked.
I mean seriously, why would an e-mail provider like this allow e-mails to be sent from "[email protected]" and "[email protected]"??
Can you elaborate how a 3rd party is able to send e-mail from @sendgrid.com?
Simple. Their ip addresses of their client/guest-servers are whitelisted in the SPF records.
So anyone can impersonate them.
Not 100% sure if that was the case now, as I can't find the email anymore, but I've been getting them from both sendgrid and mailgun.
Mailgun's SPF (was at least):
This is actually part of a bigger issue, where lots of people use the same email service to send out emails. In theory that should work with gmail or office365, but iirc gmail forces your from sometimes, and google+ms probably thought about this, right? ;)
Not 100% sure if that was the case now, as I can't find the email anymore, but I've been getting them from both sendgrid and mailgun.
Mailgun's SPF (was at least):
v=spf1 include:spf1.mailgun.org include:spf2.mailgun.org include:_spf.google.com include:aspmx.pardot.com include:mail.zendesk.com include:spf.mailjet.com ~all
spf1.mailgun.org: v=spf1 ip4:104.130.122.0/23 ip4:146.20.112.0/26 ip4:141.193.32.0/23 ip4:161.38.192.0/20 ~all
v=spf1 ip4:209.61.151.0/24 ip4:166.78.68.0/22 ip4:198.61.254.0/23 ip4:192.237.158.0/23 ip4:23.253.182.0/23 ip4:104.130.96.0/28 ip4:146.20.113.0/24 ip4:146.20.191.0/24 ip4:159.135.224.0/20 ip4:69.72.32.0/20 ~all
This is actually part of a bigger issue, where lots of people use the same email service to send out emails. In theory that should work with gmail or office365, but iirc gmail forces your from sometimes, and google+ms probably thought about this, right? ;)
I'm surprised they give clients access to their own DKIM.
It doesn't pass DKIM. But the fact that it's sent through Sendgrid's platform gives the e-mail credibility. Because no one expects Sendgrid to allow random customers to set e-mail (not necessarily envelope) from address to @sendgrid.com.
Based on my experience with sendgrid it's par for the course.
Someone sent spam emails using my domain name, so I looked a bit into it and made a summary of what I learned: https://www.usertrack.net/blog/stop-others-use-your-domain-e...
I still remember when SendGrid got DDoSed after an incident at PyCon* and they caved. This told me two things: 1) they weren’t prepared to stand up for their own employees when the chips were down 2) their engineering wasn’t good enough to cope with a bunch of amateurs attacking them.
* Let’s not derail this into a discussion of whether or not they should have done it in the first place. They didn’t, they only acted after they couldn’t cope with the DDoS.
* Let’s not derail this into a discussion of whether or not they should have done it in the first place. They didn’t, they only acted after they couldn’t cope with the DDoS.
For those of us who don't know / remember the incident: https://venturebeat.com/2013/03/21/sendgrid-under-ddos-attac...
[deleted]
My university straight up blocks any mail coming from a SendGrid IP because of the massive amount of phishing.
What would be a huge UX and security improvement is showing which team members are using 2FA. Just like GitLab shows a little blue label next to the username.
What would be a huge UX and security improvement is showing which team members are using 2FA. Just like GitLab shows a little blue label next to the username.
That's actually good. Sendgrid & the rest are excluded from the paranoid gmail, m$, etc spam filters, while small senders suffer from being sent to spam immediately.
There should be no exceptions.
There should be no exceptions.
Wait, you mean the small timers can't afford $15,000 for services like Return Path?
The entire thing is a protection racket. There are super expensive services that can get you onto those same whitelists.
The entire thing is a protection racket. There are super expensive services that can get you onto those same whitelists.
I was under the impression that individual servers should be fine as long as their ip isn't on a recognized blacklist and they use free things like SPF and DMARC - is that not true?
It still possible to send mail from a small mail server, but there no grantee at all that messages will not end up in a spam folder. Most likely Gmail uses some ML based spam filter which has relatively high rate of false positives. For big senders they can do something manually, but they don't care about smaller ones. Gmail is too big and have little incentives to improve they spam filter (until users will start to leave, which will not happen anytime soon).
It's mostly true.
So long as you do not also send spam, or send mail that someone might think is spam. I'm a mail admin and can see my users's spam reports. Lots of those messages are authenticated and there's actually not that much malicious stuff being delivered in the first place. It is just that the messages aren't wanted.
Once your messages start to get reported, the sender's reputation goes down until its spamscore qualifies for the filter.
Once your messages start to get reported, the sender's reputation goes down until its spamscore qualifies for the filter.
It's not generally true unless you're a Capital-E grade Enterprise customer running a single server as a side component of a much larger Internet presence.
Finding connectivity that's not already blacklisted by the GOOG-MSFT email protection racket and will allow individual servers to send outgoing SMTP is basically impossible for sub-enterprise scale organizations. The major cloud services block outbound SMTP for non-Enterprise customers and minor cloud services will already be blacklisted.
Getting your mail delivered by GOOG-MSFT also means that you need to have matching forward and reverse DNS, and ideally you should have your own ASN that has never been associated with any addresses that have ever been accused of sending spam. This is an impossible hurdle except for large organizations.
The entire email ecosystem is stacked for force smaller players to pay a head tax either directly to GOOG-MSFT, in the form of using their mail services and paying extortionate per-user mailbox costs, or by using paid mail relay services.
Finding connectivity that's not already blacklisted by the GOOG-MSFT email protection racket and will allow individual servers to send outgoing SMTP is basically impossible for sub-enterprise scale organizations. The major cloud services block outbound SMTP for non-Enterprise customers and minor cloud services will already be blacklisted.
Getting your mail delivered by GOOG-MSFT also means that you need to have matching forward and reverse DNS, and ideally you should have your own ASN that has never been associated with any addresses that have ever been accused of sending spam. This is an impossible hurdle except for large organizations.
The entire email ecosystem is stacked for force smaller players to pay a head tax either directly to GOOG-MSFT, in the form of using their mail services and paying extortionate per-user mailbox costs, or by using paid mail relay services.
Yes, time for GMail et al to block them or admit it's just more anti-competitive nonsense.
If you're not whitelisted there are stricter rules and some non obvious pitfalls, but small senders with correctly configured servers does not get sent to spam in my experience.
When you pay Sendgrid or one of their competitors, you pay to not spend time dealing with this.
The overall result (in general) is to make spamming harder.
When you pay Sendgrid or one of their competitors, you pay to not spend time dealing with this.
The overall result (in general) is to make spamming harder.
If you're not on the contact list of a personal gmail account, it's nearly certain you'll land in spam if you cold contact them. Happened to me so many times; sending a mail to the person sitting next to me who I hadn't ever contacted form my own mail server (dmarc, dkim, spf, even effing google mta-sts is configured) and I still landed in spam.
This is contrary to my experience.
The number one pitfall is that your server IPs are blacklisted (directly or by association). If you use the cheapest VPS providers they will mostly be completely blacklisted since spammers have previously used them.
The number one pitfall is that your server IPs are blacklisted (directly or by association). If you use the cheapest VPS providers they will mostly be completely blacklisted since spammers have previously used them.
We use our own email servers with our own IPs for automated emails (password resets, login tokens, welcome email, etc.) and MailChimp for newsletters. We have never had any real problem with Gmail or any other email provider.
We do see some of our clients having those problems, as far as we can tell, it is mostly caused by some sort of bad rating on IP blocks.
We do see some of our clients having those problems, as far as we can tell, it is mostly caused by some sort of bad rating on IP blocks.
I mainly send from my own mail server, too. If I worry that email to a cold gmail contact will end in spam, I use a simple fallback strategy. I have gmail setup to send as an alternate email address, so it appears to come from my server's domain. The contact will certainly see gmail to gmail email, and if they reply, my email address goes into their (effective) contact list.
But is sendgrid "too big to fail"?
Too big to fail also was meant to communicate the vast amount of effort, time, and money that would be involved in rebuilding the financial infrastructure if indeed it had collapse, not to mention the immense short term pain and suffering that would occur in the interim.
A SendGrid sized void could be filled in a matter of weeks to months by teams of highly motivated engineers looking to fill a market gap. The cost of starting up there is virtually nonzero (compared to say, starting a bank), and the regulatory burden is virtually nonzero (Email regulations are arguably some of the most clear and concise).
All that to say, drawing a comparison with the "too big to fail" bailout of major US lenders is, frankly, silly.
A SendGrid sized void could be filled in a matter of weeks to months by teams of highly motivated engineers looking to fill a market gap. The cost of starting up there is virtually nonzero (compared to say, starting a bank), and the regulatory burden is virtually nonzero (Email regulations are arguably some of the most clear and concise).
All that to say, drawing a comparison with the "too big to fail" bailout of major US lenders is, frankly, silly.
Not even close.
Sendgrid and its cohorts have always been a source of spam. The spamassassin setup on my postfix SMTP gateway ranks all of the bulk email newsletter sending services with +2.5 points or higher.
This is like a sewage treatment plant complaining someone has come at 3am, cut the lock off their gate and dumped a tanker truck full of additional sewage.
This is like a sewage treatment plant complaining someone has come at 3am, cut the lock off their gate and dumped a tanker truck full of additional sewage.
> Sendgrid and its cohorts have always been a source of spam.
Sendgrid is the source of a lot of e-mail spam and their parent company Twilio is the source of a lot of SMS spam. It's really a synergy match made in MBA heaven.
Sendgrid is the source of a lot of e-mail spam and their parent company Twilio is the source of a lot of SMS spam. It's really a synergy match made in MBA heaven.
Sigh. Same happened to me. A few months ago. Sendgrid account hacked, someone set up a campaign and sent a huge amount of spam mail. Big bill on my credit card. Contacted Sendgrid, they blamed it on me. I cancelled my account, never going back. Screw them
The same thing happened to my previous employer a few months ago.
Our SRE had used a very long novel password and access was very limited, but 2FA wasn't enabled because it lacked some features (I'm guessing it was the SMTP, as other poster mentioned).
We were able to catch it pretty fast and never asked for refunds or anything, but Sendgrid's support was very standoffish and combative when we asked for help with our "investigation" of how the account was compromised, to the point the company started seeking a competitor.
Our SRE had used a very long novel password and access was very limited, but 2FA wasn't enabled because it lacked some features (I'm guessing it was the SMTP, as other poster mentioned).
We were able to catch it pretty fast and never asked for refunds or anything, but Sendgrid's support was very standoffish and combative when we asked for help with our "investigation" of how the account was compromised, to the point the company started seeking a competitor.
Don't forget to charge back.
Was it not your fault?
Is that sarcasm?
Users are at least partially responsible if they use a non unique password and then get hacked.
However, passwords are a shared secret, so you actually don't necessarily know who was responsible for the secret leaking, you or the other party.
Most proposed replacements or supplements are no better except (you can guess I'm about to say) WebAuthn. WebAuthn is built out of public key signatures, so a Relying Party has no idea how to present valid credentials even for their own system. Obviously you could just add a back door if you want one, but the WebAuthn credentials you're storing aren't like a password, or a password hash, or a TOTP seed. They're just an opaque ID and a public key.
Most proposed replacements or supplements are no better except (you can guess I'm about to say) WebAuthn. WebAuthn is built out of public key signatures, so a Relying Party has no idea how to present valid credentials even for their own system. Obviously you could just add a back door if you want one, but the WebAuthn credentials you're storing aren't like a password, or a password hash, or a TOTP seed. They're just an opaque ID and a public key.
Thats very true. But i think if sendgrid leaked their password db the level of spam would be mich higher.
Of course its always possible the password db leaked but it was using argon2 or something so only really weak passwords could be decoded.
I agree webauthn is great, and its the only option (other than TLS client certs which is a usability disaster, and maybe push notice 2fa depending on how its implemented) that protect against server side breach. However the vast majority of issues are password reuse, and to a much lesser extent dictionary attacks. TOTP effectively stops those. Then again so would choosing the users passwords for them.
Of course its always possible the password db leaked but it was using argon2 or something so only really weak passwords could be decoded.
I agree webauthn is great, and its the only option (other than TLS client certs which is a usability disaster, and maybe push notice 2fa depending on how its implemented) that protect against server side breach. However the vast majority of issues are password reuse, and to a much lesser extent dictionary attacks. TOTP effectively stops those. Then again so would choosing the users passwords for them.
All the "Push notice" 2FA systems I've used or looked at are already cheerfully bypassed by systems for phishing TOTP and SMS 2FA.
That's because they're relying on a human to make a security decision and humans are terrible at this job.
The human is at fake-site.example which they think is the real site because humans are lazy and incompetent. The fake site's backend automatically calls the real site, and the real site sends the human a push notice as 2FA. "Are you signing into Real Site®?". "Actually I wasn't sure until I got this reassuring message" thinks the human, "Yes". And now the bad guys have access to their account.
There are a lot of HN stories which are dubious about AI because they suspect it can't ever match a human in terms of vague things like intuition and learning. But you know what computers are unmistakably good at? Trivial symbol matching. fake-site.example is not real-site.example, those strings are different. That's why you just can't use your real-site.example WebAuthn credentials on fake-site.example
That's because they're relying on a human to make a security decision and humans are terrible at this job.
The human is at fake-site.example which they think is the real site because humans are lazy and incompetent. The fake site's backend automatically calls the real site, and the real site sends the human a push notice as 2FA. "Are you signing into Real Site®?". "Actually I wasn't sure until I got this reassuring message" thinks the human, "Yes". And now the bad guys have access to their account.
There are a lot of HN stories which are dubious about AI because they suspect it can't ever match a human in terms of vague things like intuition and learning. But you know what computers are unmistakably good at? Trivial symbol matching. fake-site.example is not real-site.example, those strings are different. That's why you just can't use your real-site.example WebAuthn credentials on fake-site.example
This has been ongoing for months already. Nobody at Sendgrid cared. We had several issues with them leading to penalize mails from their IP space.
I’m actively discouraging my customers from using their service.
I’m actively discouraging my customers from using their service.
> ... over the past few months there has been a marked increase in malicious, phishous and outright spammy email being blasted out via Sendgrid’s servers.
Fortunately, I hadn't noticed the recent increases.
After it became apparent that they didn't bother looking ibto -- or even reading -- abuse reports, we blacklisted basically their entire IP ranges a long time ago and then added exceptions when they were requested.
I certainly did notice the marked decrease in spam then, though.
Fortunately, I hadn't noticed the recent increases.
After it became apparent that they didn't bother looking ibto -- or even reading -- abuse reports, we blacklisted basically their entire IP ranges a long time ago and then added exceptions when they were requested.
I certainly did notice the marked decrease in spam then, though.
I've been a paying customer for years on their shared server plan ($15/mo) and I've had terrible customer service and lots of emails marked as spam (e.g. to my Mom) due to a bad reputation of the shared box. I haven't migrated out of laziness and inertia.
This article and the fact that other comments mention that SendGrid is seen as spammy is the final straw and I'm going to migrate. What paid email senders do people recommend that aren't excessively marked as spammy by gmail, etc.?
This article and the fact that other comments mention that SendGrid is seen as spammy is the final straw and I'm going to migrate. What paid email senders do people recommend that aren't excessively marked as spammy by gmail, etc.?
Throwaway since I’m a former Twilion.
Even though people love Twilio for their developer friendliness and nice APIs, what no one wants to talk about publicly is their disproportionate amount of revenue that comes from spammy behavior. Either as a company or it’s customers.
I was in sales. I saw the revenue numbers and sold these deals. It’s staggering.
Yet wall street looooooves Twilio because of their revenue growth.
Note: Twilio owns Sendgrid.
Even though people love Twilio for their developer friendliness and nice APIs, what no one wants to talk about publicly is their disproportionate amount of revenue that comes from spammy behavior. Either as a company or it’s customers.
I was in sales. I saw the revenue numbers and sold these deals. It’s staggering.
Yet wall street looooooves Twilio because of their revenue growth.
Note: Twilio owns Sendgrid.
What I would like to know from a Twilion:
Is Twilio used by commercial parties in the past to participate in voting in the European Song Festival?
Is Twilio used by commercial parties in the past to participate in voting in the European Song Festival?
Did you mean the Eurovision Song Contest, or something else?
You may have luck with protonmail's paid tiers. I use the free tier currently and have been pretty happy.
Thanks but I host my own email so I'm only looking for an outbound SMTP server. I went to ProtonMail's business page and the pricing plan only shows full email hosting which I don't need.
After doing some more research, it seems the major alternatives are Postmark, MailGun, and Amazon SES. I liked one comment left by Postmark on another HN post where they say they want high quality senders, so I think I'm going to Postmark...
Same here regarding the support. I helped a company migrate to Sendgrid, and right after paying and integrating, we were locked out of the customer panel.
We created a ticket, and a few hours later we were also locked out of the support area. All in all, it took them 7 days from locking us out before we received a human response and they fixed it.
We recently couldn't deliver to Hotmail. It took them 27 days before the first response (P2 Degraded). They didn't fix it, and wanted us to switch plans.
We created a ticket, and a few hours later we were also locked out of the support area. All in all, it took them 7 days from locking us out before we received a human response and they fixed it.
We recently couldn't deliver to Hotmail. It took them 27 days before the first response (P2 Degraded). They didn't fix it, and wanted us to switch plans.
Are you still using them? Why?
We currently still have a subscription, but the project we're using it for became on hold during the Hotmail problem. The project will be cancelled soon, and we'll cancel our Sendgrid subscription.
If the project wouldn't have been cancelled, we would've probably switched to another email service.
If the project wouldn't have been cancelled, we would've probably switched to another email service.
I've also experienced my IPs getting blacklisted (albeit on the free plan). I can understand this happening (you get what you pay for) but what I can't understand is that their support response was saying that I just had to deal with it rather than switching my IP. I've also used MailGun and not had that problem (although I'm not a fan of the way you can't export logs from them for some reason).
how I hate this kind of reporting
3 people complain on reddit and it's now "[COMPANY X] under siege"
3 people complain on reddit and it's now "[COMPANY X] under siege"
Between the comments on the article itself and this HN thread, there's already more than 10 times that. It's not being exaggerated quite the way that you "hate".
To further prove how bad SendGrid is, Invaluement recently spent a ton of time and energy to create an entirely new "Service Provider DNSBL" [0] -- just for SendGrid! In fact, they just announced it about a week ago [1].
[0]: https://www.invaluement.com/serviceproviderdnsbl/
[1]: https://www.mail-archive.com/[email protected]/msg11403.html
To further prove how bad SendGrid is, Invaluement recently spent a ton of time and energy to create an entirely new "Service Provider DNSBL" [0] -- just for SendGrid! In fact, they just announced it about a week ago [1].
[0]: https://www.invaluement.com/serviceproviderdnsbl/
[1]: https://www.mail-archive.com/[email protected]/msg11403.html
Spend a few minutes looking at the spam list threads linked in the article. This is not just a few people complaining. E.g.:
https://www.mail-archive.com/search?l=mailop%40mailop.org&q=...
There was a ton of material I did not include in the story, including a story from a company that had a client have 40 million phishing emails sent through their Sendgrid account, which it turns out was set up by an employee long ago who was no longer with the company and had not turned on 2FA (and probably was re-using passwords).
I started reporting this story almost a month ago after receiving more than 3 emails from different IT experts who were really frustrated with the amount of malware and phishing coming from Sendgrid accounts. They were frustrated because they couldn't block Sendgrid outright because too many companies they were expecting regular emails from used the platform.
The day before I published the story, I head from someone else who was getting phishing attacks spoofing Aruba Networks.
https://www.mail-archive.com/search?l=mailop%40mailop.org&q=...
There was a ton of material I did not include in the story, including a story from a company that had a client have 40 million phishing emails sent through their Sendgrid account, which it turns out was set up by an employee long ago who was no longer with the company and had not turned on 2FA (and probably was re-using passwords).
I started reporting this story almost a month ago after receiving more than 3 emails from different IT experts who were really frustrated with the amount of malware and phishing coming from Sendgrid accounts. They were frustrated because they couldn't block Sendgrid outright because too many companies they were expecting regular emails from used the platform.
The day before I published the story, I head from someone else who was getting phishing attacks spoofing Aruba Networks.
You also could have a look on twitter how many people with hacked accounts try to reach sendgrid support via this channel.
The CSO said:
> "... 2FA for customer accounts is the right thing to do, and we’re working towards that end ... This is part of the reason we acquired Authy ..."
Cool, it sounds like they're making a effort to implement -- and require -- 2FA... oh wait, FIVE YEARS have passed since then!
> "... 2FA for customer accounts is the right thing to do, and we’re working towards that end ... This is part of the reason we acquired Authy ..."
Cool, it sounds like they're making a effort to implement -- and require -- 2FA... oh wait, FIVE YEARS have passed since then!
Twilio supports and soon will require TFA. I personally don't use SendGrid (use Mailgun), but I have to believe it should be coming soon (if not already supported).
see my comment here: https://news.ycombinator.com/item?id=24319007
If you run a service that can be abused like this, you're going to need objective account abuse metrics to warn you before these things happen. Specifically you're going to need a Russian speaker to prowl around on Russian forums buying your own hacked accounts, so you can figure out the market price. If accounts on your system are selling for ten cents, might be time to reflect on account security.
I run IT for my organization and I am seeing a lot of phishing emails from Sendgrid. I haven't counted or done data analysis for proportion, but it's enough volume for me to really notice it.
This list is probably incomplete / out-of-date but ...
149.72.0.0/16
167.89.0.0/17
168.245.0.0/17
192.254.112.0/20
198.21.0.0/21
198.37.144.0/20Isn't their SPF record authoritative?
ip4:167.89.0.0/17 ip4:208.117.48.0/20 ip4:50.31.32.0/19 ip4:198.37.144.0/20 ip4:198.21.0.0/21 ip4:192.254.112.0/20 ip4:168.245.0.0/17 ip4:149.72.0.0/16
ip4:167.89.0.0/17 ip4:208.117.48.0/20 ip4:50.31.32.0/19 ip4:198.37.144.0/20 ip4:198.21.0.0/21 ip4:192.254.112.0/20 ip4:168.245.0.0/17 ip4:149.72.0.0/16
Usually, yeah, but not necessarily (e.g., for those with dedicated IP addresses).
Even then, it usually will be since most customers will just add the (customer-specific) "include:nnn.wl.sendgrid.net", or even just "include:sendgrid.net". But, for whatever reason, I've come across several that add an "a:" or "ip4:" for their dedicated IPs instead (I assume there's a good reason for that, but I don't know what it is).
Even then, it usually will be since most customers will just add the (customer-specific) "include:nnn.wl.sendgrid.net", or even just "include:sendgrid.net". But, for whatever reason, I've come across several that add an "a:" or "ip4:" for their dedicated IPs instead (I assume there's a good reason for that, but I don't know what it is).
Probably so that they can also send some mail from their own servers, in addition to sendgrid's.
Sendgrid user for many, many years.
I believe we disabled 2FA because it broke /prevented our client's ability to send via SMTP. No idea if they've fixed the issue (and honestly don't care). Make security easy to adopt of you want us to implement it.
Also, organization-wide, we don't use 2FA that requires a mobile phone number/SMS (because we don't trust vendors with our phone number).
*IIRC, Sendgrid initially only offered 2FA via SMS, but now you can also use an authenticator app.
I believe we disabled 2FA because it broke /prevented our client's ability to send via SMTP. No idea if they've fixed the issue (and honestly don't care). Make security easy to adopt of you want us to implement it.
Also, organization-wide, we don't use 2FA that requires a mobile phone number/SMS (because we don't trust vendors with our phone number).
*IIRC, Sendgrid initially only offered 2FA via SMS, but now you can also use an authenticator app.
Phone 2FA is broken by default.
Yes, it broke SMTP and it also broke all API access until they kinda-fixed API keys. Pretty much anything you used basic auth for at a touch point.
Yes, it broke SMTP and it also broke all API access until they kinda-fixed API keys. Pretty much anything you used basic auth for at a touch point.
> Neil Schwartzman, executive director of the anti-spam group CAUCE, said Sendgrid’s 2FA plans are long overdue, noting that the company bought Authy back in 2015.
It's worth noting that this mixes up history. Authy was acquired by Twilio in 2015. Sendgrid was acquired by Twilio in 2019, there's no reason to assume in the 4 years interim that SG should have used Authy.
It's worth noting that this mixes up history. Authy was acquired by Twilio in 2015. Sendgrid was acquired by Twilio in 2019, there's no reason to assume in the 4 years interim that SG should have used Authy.
I think the point was that they own a literal 2FA company, but don't have 2FA
For a company that went out and acquired a 2FA subsidiary, Twilio really doesn't seem to have their 2FA story together.
For example you'd think a company with an division dedicated to authentication would support what the industry considers the best 2FA solution: WebAuthn. Sadly that is not an option for your Twilio account (other Auth providers like Duo support WebAuthn).
Twilio has also been emailing me the past few months telling me I must turn on 2FA on my account. This is weird since I have TOTP enabled.
I feel like Twilio must be internally conflicted about this. As a company dedicated to phone services they really want to push SMS solutions even though SMS based 2FA is vastly inferior to WebAuthn.
For example you'd think a company with an division dedicated to authentication would support what the industry considers the best 2FA solution: WebAuthn. Sadly that is not an option for your Twilio account (other Auth providers like Duo support WebAuthn).
Twilio has also been emailing me the past few months telling me I must turn on 2FA on my account. This is weird since I have TOTP enabled.
I feel like Twilio must be internally conflicted about this. As a company dedicated to phone services they really want to push SMS solutions even though SMS based 2FA is vastly inferior to WebAuthn.
> Twilio has also been emailing me the past few months telling me I must turn on 2FA on my account. This is weird since I have TOTP enabled.
There's been a bit of confusion about this email: you can turn on 2FA separately for your Twilio account (identified by a 32-char hex string prefixed with "AC") and your user account (your email address). The email is talking about your user account. Even if you have 2FA turned on for your Twilio account, the email is asking you to turn it on for your user account.
For some context for those who may not be familiar: the Twilio account is essentially a billing/project unit that is a container for Twilio resources you've purchased or configured. Multiple user accounts can have access to a single Twilio account, and a single user account can have access to multiple Twilio accounts. Enabling 2FA on the Twilio account means all users who want to sign into that account (regardless of their user account setting) will have to use 2FA. Similarly, enabling 2FA on your user account will require 2FA whenever you sign in, regardless of the settings on any accounts you may have access to.
(Source: I work there, got confused about this email myself, and managed to clarify internally what was going on.)
There's been a bit of confusion about this email: you can turn on 2FA separately for your Twilio account (identified by a 32-char hex string prefixed with "AC") and your user account (your email address). The email is talking about your user account. Even if you have 2FA turned on for your Twilio account, the email is asking you to turn it on for your user account.
For some context for those who may not be familiar: the Twilio account is essentially a billing/project unit that is a container for Twilio resources you've purchased or configured. Multiple user accounts can have access to a single Twilio account, and a single user account can have access to multiple Twilio accounts. Enabling 2FA on the Twilio account means all users who want to sign into that account (regardless of their user account setting) will have to use 2FA. Similarly, enabling 2FA on your user account will require 2FA whenever you sign in, regardless of the settings on any accounts you may have access to.
(Source: I work there, got confused about this email myself, and managed to clarify internally what was going on.)
> even though SMS based 2FA is vastly inferior to WebAuthn.
SMS based 2FA is worse than useless thanks to the rise of SIM Swapping hacks, it adds an easy attack vector into your account.
SMS based 2FA is worse than useless thanks to the rise of SIM Swapping hacks, it adds an easy attack vector into your account.
> you'd think a company with an division dedicated to authentication would support what the industry considers the best 2FA solution: WebAuthn
How would WebAuthn work with someone using a mail user agent connecting via SMTP?
How would WebAuthn work with someone using a mail user agent connecting via SMTP?
What Sendgrid offers here (and this is fairly typical) goes like this:
* To use their APIs or SMTP submission servers you need a bearer token, which is basically a random blob of data.
* To get a new bearer token (good for any number of API calls or SMTP submissions) you log into a web site and request a new token. This site is also where you can de-authorize existing tokens. The site is protected with 2FA
Today, Sendgrid offers this, only with Authy for 2FA and it's optional. If you decide bearer tokens are too complicated for your 15 year old PHP mail sending code, you can just use a username and self-selected password for SMTP or the API instead.
Authy has an obligatory SMS bypass. So even though you can use an app to generate codes, bad guys who can SIM swap their way to your phone number can do 2FA and get into the web site to issue their own bearer tokens.
So, today if you can guess a company's username and password on Sendgrid there's a good chance that's enough to have Sendgrid help you send spam as that company.
With the 2FA world they want to get to, you would need to either SIM-swap, trick their customer service agents, or most likely just pinch a bearer token they wrote to a Pastebin or whatever.
They could do much better in 2020, but there's no sign Sendgrid has any interest in doing more than the very bare minimum.
* To use their APIs or SMTP submission servers you need a bearer token, which is basically a random blob of data.
* To get a new bearer token (good for any number of API calls or SMTP submissions) you log into a web site and request a new token. This site is also where you can de-authorize existing tokens. The site is protected with 2FA
Today, Sendgrid offers this, only with Authy for 2FA and it's optional. If you decide bearer tokens are too complicated for your 15 year old PHP mail sending code, you can just use a username and self-selected password for SMTP or the API instead.
Authy has an obligatory SMS bypass. So even though you can use an app to generate codes, bad guys who can SIM swap their way to your phone number can do 2FA and get into the web site to issue their own bearer tokens.
So, today if you can guess a company's username and password on Sendgrid there's a good chance that's enough to have Sendgrid help you send spam as that company.
With the 2FA world they want to get to, you would need to either SIM-swap, trick their customer service agents, or most likely just pinch a bearer token they wrote to a Pastebin or whatever.
They could do much better in 2020, but there's no sign Sendgrid has any interest in doing more than the very bare minimum.
> To get a new bearer token (good for any number of API calls or SMTP submissions) you log into a web site and request a new token.
Is the bearer token used as the password in the SMTP transaction? Could the same one be used for IMAP access?
Is the bearer token used as the password in the SMTP transaction? Could the same one be used for IMAP access?
I don’t get this sense at all... as a long time Twilio customer... they have since on the Authy purchase been pretty consistently pushing it as the solution for two factor... sms is a medium for retrieving the 2nd code by not pushed in anyway...
They don't seem that conflicted to me. I have always assumed their SMS-only 2fa was an attempt to sell people on SMS as 2fa. But all I want is to use my Yubico.
What service do people recommend in place of SendGrid?
Personally I need a service to send transactional emails from my company website with a low chance that they’ll be marked as spam. Tracking is not necessary (I disabled that in SendGrid).
Personally I need a service to send transactional emails from my company website with a low chance that they’ll be marked as spam. Tracking is not necessary (I disabled that in SendGrid).
Amazon SES can work for that.
And depending how much you need to send, if you’re already in AWS the first 62k messages every month are free.
The only annoying thing is their sandbox and usage limits. You’re supposed to accept a lower limit, send high quality mail, and then request a higher limit once they’ve seen that you’re not generating a bunch of abuse reports.
Slowly on boarding senders is good for delivery but makes it difficult to just do a simple cut-over if you’re currently sending more than whatever quota they’ll give you right now. You need to continue sending through your old service and slowly move your usage over.
The only annoying thing is their sandbox and usage limits. You’re supposed to accept a lower limit, send high quality mail, and then request a higher limit once they’ve seen that you’re not generating a bunch of abuse reports.
Slowly on boarding senders is good for delivery but makes it difficult to just do a simple cut-over if you’re currently sending more than whatever quota they’ll give you right now. You need to continue sending through your old service and slowly move your usage over.
Maybe Twilio will use that run up (bubble) in stock price to allocate to some tech debt like this.
169% YTD 33.8% last 3 months
I’ve always looked at SendGrid as a premium provider for transactional mail. Why recommendations for something better but similarly priced?
169% YTD 33.8% last 3 months
I’ve always looked at SendGrid as a premium provider for transactional mail. Why recommendations for something better but similarly priced?
I developed a tool called shhgit.com that watches code commits across GitHub, Gitlab and Bitbucket in real time. In a ~24 hour period you will find 100s of valid SendGrid credentials/API keys so this does not surprise me. They really need to enforce 2FA and IP whitelisting for API key use.
Oof. So,I tried sending mails from our own domain and our mail server - but Google tagged us as spam. We migrated into Google hosting and it was fixed, now back to Spam. So we move to a provider like sendgrid, but then they have a hack. Damnit!
When we hosted our own, we had DKIM, SPF, DMARC and such, clean IP and clean messages, we only sent account create and transaction notices - low volume, relevant messages to folks who opt-in. We had our little box setup right and we're polite actors but got tagged which killed our user experience (password resets went to Spam and folk called support)
But it feels the "big-web" forced us into vulnerable providers for the spam game.
We need more self hosted email and less centralized control.
When we hosted our own, we had DKIM, SPF, DMARC and such, clean IP and clean messages, we only sent account create and transaction notices - low volume, relevant messages to folks who opt-in. We had our little box setup right and we're polite actors but got tagged which killed our user experience (password resets went to Spam and folk called support)
But it feels the "big-web" forced us into vulnerable providers for the spam game.
We need more self hosted email and less centralized control.
`We need more self hosted email and less centralized control`
We need to start over and build a truly open internet.
We need to start over and build a truly open internet.
> We need to start over and build a truly open internet.
I want to agree, but email is actually an example of an open and federated protocol achieving success. It's pretty much everything that could have been hoped for as far as openness and interoperability go at the protocol level. Unfortunately, balkanization still occurs at the social (ie network) level.
I want to agree, but email is actually an example of an open and federated protocol achieving success. It's pretty much everything that could have been hoped for as far as openness and interoperability go at the protocol level. Unfortunately, balkanization still occurs at the social (ie network) level.
The open email Internet was buried in an avalanche of spam.
Open alone is insufficient.
Open alone is insufficient.
[deleted]
If email were architected to put the cost of storage on senders then I think the spam problem would have been more manageable. Instead it's so closely modeled on physical mail that recipients get overwhelmed quickly.
Even if email worked like that, I don't think it would solve the spam problem. Spam emails are rarely personalized and the same exact message is sent out to many different addresses.
As such, the spammer would only have to store each unique message and the list of recipients associated with each unique message.
As such, the spammer would only have to store each unique message and the list of recipients associated with each unique message.
Even with single copy + list, with a poll-and-request delivery model (sender polls recipient's system, which then requests message on some schedule):
1. Sender canot simply mass-cram messages to recipients, as many spam engines today do. (Qmail for a time was notorious for doing this).
2. There's good chance that any bulk spammer will accrue a negative reputation by the time most recipients bother to request messages. So they won't.
3. Spammer must sustain a stable online presence --- not be knocked offline or blocked by ISP or hosting provider.
4. Spammer has to track successful deliveries vs. messages that are complete.
Upshot: the spammer's workfactor increases.
1. Sender canot simply mass-cram messages to recipients, as many spam engines today do. (Qmail for a time was notorious for doing this).
2. There's good chance that any bulk spammer will accrue a negative reputation by the time most recipients bother to request messages. So they won't.
3. Spammer must sustain a stable online presence --- not be knocked offline or blocked by ISP or hosting provider.
4. Spammer has to track successful deliveries vs. messages that are complete.
Upshot: the spammer's workfactor increases.
Hrm. I wonder if this is an actual use case for blockchain and proof of work.
If you want to send me an email, you have to send me an SMTPCoin. If my user doesn't mark it as spam, I give you the coin back. If my user marks it as spam, I keep the coin.
If you want to send me an email, you have to send me an SMTPCoin. If my user doesn't mark it as spam, I give you the coin back. If my user marks it as spam, I keep the coin.
Preventing email spam was in fact the original use case for proof of work.
https://en.wikipedia.org/wiki/Hashcash
https://en.wikipedia.org/wiki/Hashcash
This was the inspirstion for nano's use of proof of work too. They don't use PoW for the chain itself but do require every transaction to include a small proof of work to avoid spamming the network, almost identical to hashcash.
We need more self hosted email and less centralized control.
This is true, from both sides of the equation. The biggest single problem with email today, IME at least, is the number of major mail providers who are acting as gatekeepers for their users and doing a bad job of it. In particular, they block way too much legitimate mail, and often do it silently, so the sender is not even aware of the problem or able to contact them about getting it fixed.
Then the sender of the missing mail gets the customer support requests about the missing password reset emails, or the complaints that someone didn't know they were still subscribed despite the receipt emails being sent for each payment, or...
At this point, there really ought to be a blacklist for unreliable mail services on the receiving side analogous to the spam blacklists, so businesses can warn their users if given an address on a bad service and invite them to choose another.
This is true, from both sides of the equation. The biggest single problem with email today, IME at least, is the number of major mail providers who are acting as gatekeepers for their users and doing a bad job of it. In particular, they block way too much legitimate mail, and often do it silently, so the sender is not even aware of the problem or able to contact them about getting it fixed.
Then the sender of the missing mail gets the customer support requests about the missing password reset emails, or the complaints that someone didn't know they were still subscribed despite the receipt emails being sent for each payment, or...
At this point, there really ought to be a blacklist for unreliable mail services on the receiving side analogous to the spam blacklists, so businesses can warn their users if given an address on a bad service and invite them to choose another.
At this point in time I think I'd rather deal with the spammers than with Google, Microsoft etc. Unfortunately they've managed to lock up email to all intents and purposes, good luck reversing the situation to the point where they have to play nice.
To be honest, at this point I wish email would just die and get replaced with an actual modern protocol. It's a nightmare. Everything about it sucks.
Please don't. Anything modern would another lock-in. I can read decade old emails, switched multiple providers. Messages in those 'new protocols' are long unaccessible and protocols themselves are dead.
Such as? How would it work?
Not having trivially forged headers would be a start.
Isn’t this basically what SPF, DKIM and SMTP over TLS get you? Sure you can try to forge messages but they should bounce immediately and not even reach the recipients inbox.
I’m not an expert... but From what I can tell no one trusts anyone’s email anymore and even as we added these new security things.. we didn’t start trusting any more than we did before... Consequences being that even with the extra security, trying to run your own mail server and get email recipients to receive the mail you send is like being some kind of extremely polite red team, on the offensive in the worlds longest least organised capture the flag.
There are plenty of people successfully hosting their own email. If you're getting flagged as spam, then you need to take a few steps every once in a while but self-hosted email is still totally doable.
As somebody who is running their own mail sever I'd love to agree with you but GMail, Hotmail, and all the other large email providers make it extremely hard to do so.
If you only send a few emails per month, you are fine. When you host an email server for others, even if it's only your family and friends, it can quickly become a nightmare. And if I ran a business, I'd never do it myself.
Landing on a blacklist of those large providers effectively shuts your email server down and it's way too easy to get blacklisted. There are so many techs around email now (DKIM, TLS, SPF, HELO testing), that coordinating them becomes a huge responsibility. Misconfiguring a single one can lead to rejection at the large providers that deal with billions of incoming emails a day, with hyper aggressive spam filters. The worst thing that can happen is that your emails were to be classified as spam but you didn't know it. A flat out rejection is relatively rare so you are often not notified that your configuration is incorrect.
Getting off those blacklists is extremely hard. Especially if you don't land on GMail's blacklist but on those community driven ones that only allow your IP to be removed from it once. If you land on them a second time, you need a new IP and hope that this IP isn't already blacklisted.
Many large providers are blacklisting entire ISP IP ranges due to past so waves. E.g. GMail is blocking the entire AWS EC2 range, I believe.
If you only send a few emails per month, you are fine. When you host an email server for others, even if it's only your family and friends, it can quickly become a nightmare. And if I ran a business, I'd never do it myself.
Landing on a blacklist of those large providers effectively shuts your email server down and it's way too easy to get blacklisted. There are so many techs around email now (DKIM, TLS, SPF, HELO testing), that coordinating them becomes a huge responsibility. Misconfiguring a single one can lead to rejection at the large providers that deal with billions of incoming emails a day, with hyper aggressive spam filters. The worst thing that can happen is that your emails were to be classified as spam but you didn't know it. A flat out rejection is relatively rare so you are often not notified that your configuration is incorrect.
Getting off those blacklists is extremely hard. Especially if you don't land on GMail's blacklist but on those community driven ones that only allow your IP to be removed from it once. If you land on them a second time, you need a new IP and hope that this IP isn't already blacklisted.
Many large providers are blacklisting entire ISP IP ranges due to past so waves. E.g. GMail is blocking the entire AWS EC2 range, I believe.
Keep in mind that some users almost certainly believe your email is spam. Even if they're explicitly paying you to send them that email.
For an end user "spam" is "email I don't want".
"But" I hear you screaming, "That's their credit card bill, they even confirmed their email address and specifically opted into receiving it by email - that's not spam". It's unwanted, they don't have $850 to pay off the balance and they don't want to think about the interest charges, so it's spam as far as the user is concerned, some non-trivial fraction of users will click "Spam".
You've probably seen people who have piles of unopened real world postal mail, they feel the same way about that. Unsolicited catalogue. Bank statement showing I am overdrawn. Nagging letter from my mother. All unwanted: Spam.
Worse, some of these same users will anyway be angry that the important message they marked as spam is treated as spam. Why are facts true? They shouldn't be true if I don't want them to be. Is that irrational? Sure, that's humans for you.
This is a losing proposition, and the end-user facing providers are aware of that, but "spam blocking" is something users demand from such providers even while being very angry that it doesn't do what they want. "Humans are irrational" is not actionable.
For an end user "spam" is "email I don't want".
"But" I hear you screaming, "That's their credit card bill, they even confirmed their email address and specifically opted into receiving it by email - that's not spam". It's unwanted, they don't have $850 to pay off the balance and they don't want to think about the interest charges, so it's spam as far as the user is concerned, some non-trivial fraction of users will click "Spam".
You've probably seen people who have piles of unopened real world postal mail, they feel the same way about that. Unsolicited catalogue. Bank statement showing I am overdrawn. Nagging letter from my mother. All unwanted: Spam.
Worse, some of these same users will anyway be angry that the important message they marked as spam is treated as spam. Why are facts true? They shouldn't be true if I don't want them to be. Is that irrational? Sure, that's humans for you.
This is a losing proposition, and the end-user facing providers are aware of that, but "spam blocking" is something users demand from such providers even while being very angry that it doesn't do what they want. "Humans are irrational" is not actionable.
> For an end user "spam" is "email I don't want".
last time I checked (been a few years since I used Google) BigTech is perfectly able to distinguish a bad actor from an opt-in type of service. They even presented me with a little notification whenever I moved stuff into SPAM that didn't belong there asking "do you want Gmail to unsubscribe you from X" ... I can't exactly recall the text - but all their lofty claims about their "AI" should account for something?
I agree with you that the _user_ is the biggest wildcard. But it's not that we should allow Google/Microsoft get way with this as an excuse. They are currently right in the middle of killing open protocols like imap in favor of their own flavored shit (probably something AMP like but for email). They do not deserve the benefit of a doubt. They do not even deserve to have their side of the argument heard at this stage.
last time I checked (been a few years since I used Google) BigTech is perfectly able to distinguish a bad actor from an opt-in type of service. They even presented me with a little notification whenever I moved stuff into SPAM that didn't belong there asking "do you want Gmail to unsubscribe you from X" ... I can't exactly recall the text - but all their lofty claims about their "AI" should account for something?
I agree with you that the _user_ is the biggest wildcard. But it's not that we should allow Google/Microsoft get way with this as an excuse. They are currently right in the middle of killing open protocols like imap in favor of their own flavored shit (probably something AMP like but for email). They do not deserve the benefit of a doubt. They do not even deserve to have their side of the argument heard at this stage.
GMail prompting you to unsubscribe is because the email carries a standard [1] header (List-Unsubscribe) and GMail detects it. They may also be using some smarts to detect unsubscribable emails that don't have the header, but in general it's because of the header. It doesn't have anything to do with distinguishing between spam and legitimate subscriptions.
So if you mark a legitimate subscription as spam and don't unsubscribe when gmail prompts you to, it's likely that it'll get sent to your spam in the future, and perhaps contribute to the sender's spam score which affects other people's inboxes too.
[1]: de-facto standard? I can't find any RFC for it.
So if you mark a legitimate subscription as spam and don't unsubscribe when gmail prompts you to, it's likely that it'll get sent to your spam in the future, and perhaps contribute to the sender's spam score which affects other people's inboxes too.
[1]: de-facto standard? I can't find any RFC for it.
actually I wasn't talking about mailing lists (I'd never move something from majordomo or similar to spam since I 100% know why I received them). But I'm talking about corporate news letters delivered by services like MailChimp and other marketing tech. - No idea if these have a List-Unsubscribe header - but even if they do I shouldn't have received the message in the first place (in most cases).
I am not a Sendgrid customer, however examining their documentation what I think the "attacks" look like is this:
14 year old Jimmy guesses that Foo Corp has the Sendgrid username "FooCorp" and the password "FooCorp0" obeying a company requirement to use "At least 8 characters" and "At least one digit". Very secure. Much password.
Jimmy connects to Sendgrid's old API and literally just provides username "FooCorp" password "FooCorp0" to send as much email as Foo Corp have paid to be able to send. No other steps needed.
Insisting customers get 2FA would in practice mean:
1. Get all customers to sign up to Authy's phone-based 2FA and use it for Sendgrid. Accept that this will have a revenue impact, you're essentially firing customers who either were too lazy to sign up for this or regard this type of security as a joke.
2. Deprecate older APIs that allow username + password auth
3. Thus force users into APIs that rely on a bearer token they can only get using 2FA.
And yet after all this work if Jimmy instead gets a copy of your bearer token (e.g. because some idiot commits it to github) then he can once again send emails as Foo Corp until somebody disables the stolen bearer token (and then generates a new one and updates all their running code).
It's a poor show that in 2020 this is their goal, it's not even the thing they have and need to improve upon, it's just a future goal.
14 year old Jimmy guesses that Foo Corp has the Sendgrid username "FooCorp" and the password "FooCorp0" obeying a company requirement to use "At least 8 characters" and "At least one digit". Very secure. Much password.
Jimmy connects to Sendgrid's old API and literally just provides username "FooCorp" password "FooCorp0" to send as much email as Foo Corp have paid to be able to send. No other steps needed.
Insisting customers get 2FA would in practice mean:
1. Get all customers to sign up to Authy's phone-based 2FA and use it for Sendgrid. Accept that this will have a revenue impact, you're essentially firing customers who either were too lazy to sign up for this or regard this type of security as a joke.
2. Deprecate older APIs that allow username + password auth
3. Thus force users into APIs that rely on a bearer token they can only get using 2FA.
And yet after all this work if Jimmy instead gets a copy of your bearer token (e.g. because some idiot commits it to github) then he can once again send emails as Foo Corp until somebody disables the stolen bearer token (and then generates a new one and updates all their running code).
It's a poor show that in 2020 this is their goal, it's not even the thing they have and need to improve upon, it's just a future goal.
The whole idea behind Sendgrid is a joke.
It's become popular for some providers since the move to the Cloud to block outgoing port 25, but apart from that, you literally don't need Sendgrid. Any senior engineer can write a local "Sendgrid" in about half a day. Sendgrid doesn't even support all the RFCs properly, so, you'll likely get dropped mail if you move to Sendgrid from Postfix/Sendmail. Besides that, Sendgrid's UI/UX was terrible the last time I've played with it. It's literally the lamest startup there could possibly be.
As a customer of some unrelated service, I've already converted said service away from Sendgrid in just a couple of emails to the owner. (Sadly, they've moved to one of the many Sendgrid's competitors, not to Postfix/Sendmail, but the switch was obviously painless.) That's how horrible Sendgrid is, and how easy it is to make other people switch.
P.S. I often send mail through shell scripts. Try selling me Sendgrid when my laptop already comes with Postfix/Sendmail, with spool support. Does Sendgrid even work when you have no internet connection? (Rhetorical question: no, Sendgrid does not when internet is not available; but Postfix/Sendmail works just fine even without any internet, and will spool all your mail until you get connected to an internet with port 25 not blocked.) Sendgrid is a downgrade however you look at it.
It's become popular for some providers since the move to the Cloud to block outgoing port 25, but apart from that, you literally don't need Sendgrid. Any senior engineer can write a local "Sendgrid" in about half a day. Sendgrid doesn't even support all the RFCs properly, so, you'll likely get dropped mail if you move to Sendgrid from Postfix/Sendmail. Besides that, Sendgrid's UI/UX was terrible the last time I've played with it. It's literally the lamest startup there could possibly be.
As a customer of some unrelated service, I've already converted said service away from Sendgrid in just a couple of emails to the owner. (Sadly, they've moved to one of the many Sendgrid's competitors, not to Postfix/Sendmail, but the switch was obviously painless.) That's how horrible Sendgrid is, and how easy it is to make other people switch.
P.S. I often send mail through shell scripts. Try selling me Sendgrid when my laptop already comes with Postfix/Sendmail, with spool support. Does Sendgrid even work when you have no internet connection? (Rhetorical question: no, Sendgrid does not when internet is not available; but Postfix/Sendmail works just fine even without any internet, and will spool all your mail until you get connected to an internet with port 25 not blocked.) Sendgrid is a downgrade however you look at it.
You can set up Sendgrid as a Postfix or Sendmail relay host.
Any sort of fixed token without expiry is poor security .
No security protocol is perfect, however using a private key aws style , or oauth2 short expiry token like google , GitHub and many others is vastly better than Basic auth like system sendgrid is using in the older API.
There are many ways without changing API you could have improved security, Rate limit , IP whitelist, force frequent password changes, force use pass phrase etc and other techniques could have mitigated this issue
No security protocol is perfect, however using a private key aws style , or oauth2 short expiry token like google , GitHub and many others is vastly better than Basic auth like system sendgrid is using in the older API.
There are many ways without changing API you could have improved security, Rate limit , IP whitelist, force frequent password changes, force use pass phrase etc and other techniques could have mitigated this issue
The whole idea behind Sendgrid is a joke.
It's become popular for some providers since the move to the Cloud to block outgoing port 25, but apart from that, you literally don't need Sendgrid. Any senior engineer can write a local "Sendgrid" in about half a day, if not less. Sendgrid doesn't even support all the RFCs properly, so, you'll likely get dropped mail if you move to Sendgrid from Postfix/Sendmail. Besides that, Sendgrid's UI/UX was terrible the last time I've played with it. It's literally the lamest startup there could possibly be.
As a customer of some unrelated service, I've already converted said service away from Sendgrid in just a couple of emails to the owner. (Sadly, they've moved to one of the many Sendgrid's competitors, not to Postfix/Sendmail, but the switch was obviously painless, and actually improved reliability.) That's how horrible Sendgrid is, and how easy it is to make other people switch.
P.S. I often send mail through shell scripts. Try selling me Sendgrid when my laptop already comes with Postfix/Sendmail, with spool support. Does Sendgrid even work when you have no internet connection? (That was a rhetorical question; the answer is, No, Sendgrid does not work when internet is not available; but Postfix/Sendmail works just fine even without any internet at all, and Postfix/Sendmail will spool all your mail until you get connected to an internet with port 25 not blocked.) Sendgrid is a downgrade however you look at it. When they move to MFA, it'll probably be easier to just switch to one of their plentiful competitors than deal with any new restrictions; that's why Sendgrid will probably be very careful around any new requirements; because their service is 100% disposable.
It's become popular for some providers since the move to the Cloud to block outgoing port 25, but apart from that, you literally don't need Sendgrid. Any senior engineer can write a local "Sendgrid" in about half a day, if not less. Sendgrid doesn't even support all the RFCs properly, so, you'll likely get dropped mail if you move to Sendgrid from Postfix/Sendmail. Besides that, Sendgrid's UI/UX was terrible the last time I've played with it. It's literally the lamest startup there could possibly be.
As a customer of some unrelated service, I've already converted said service away from Sendgrid in just a couple of emails to the owner. (Sadly, they've moved to one of the many Sendgrid's competitors, not to Postfix/Sendmail, but the switch was obviously painless, and actually improved reliability.) That's how horrible Sendgrid is, and how easy it is to make other people switch.
P.S. I often send mail through shell scripts. Try selling me Sendgrid when my laptop already comes with Postfix/Sendmail, with spool support. Does Sendgrid even work when you have no internet connection? (That was a rhetorical question; the answer is, No, Sendgrid does not work when internet is not available; but Postfix/Sendmail works just fine even without any internet at all, and Postfix/Sendmail will spool all your mail until you get connected to an internet with port 25 not blocked.) Sendgrid is a downgrade however you look at it. When they move to MFA, it'll probably be easier to just switch to one of their plentiful competitors than deal with any new restrictions; that's why Sendgrid will probably be very careful around any new requirements; because their service is 100% disposable.
You seem to not understand the sendgrid service at all. They aren't selling SMTP service, they are selling deliverability (along with tracking and metrics)
You seem to not understand the actual problem at all if you think you will be able to successfully deliver email sent directly from a local sendmail client on any random IP that lets you connect over port 25. No major email provider will accept your email coming from a residential IP.
You seem to not understand the actual problem at all if you think you will be able to successfully deliver email sent directly from a local sendmail client on any random IP that lets you connect over port 25. No major email provider will accept your email coming from a residential IP.
If that's what they're selling, they're not very good at it. Sendgrid doesn't support all applicable RFCs, and they don't support spool, so, they have exactly 0% deliverability if the recipient host is down for any reason, or uses greylisting.
I've never had any issues with mail from my WiFi IP accepted. You'd normally send it just to your own server from your local laptop w/o any extra configuration, so, it's up to you how you configure your own server. I just tried sending to Gmail directly from Postfix on my MacBook, without any extra configuration, and it's been accepted by Gmail as well. Zero configuration on the laptop. Postfix and the mail CLI clients come preinstalled, and the mail just works.
I've never had any issues with mail from my WiFi IP accepted. You'd normally send it just to your own server from your local laptop w/o any extra configuration, so, it's up to you how you configure your own server. I just tried sending to Gmail directly from Postfix on my MacBook, without any extra configuration, and it's been accepted by Gmail as well. Zero configuration on the laptop. Postfix and the mail CLI clients come preinstalled, and the mail just works.
You're mainly paying to deal with deliverability (IP reputation, proper "ramping" of mail servers, and the increasing number of technical requirements for correctly operating a mail server).
At work we had a Sendgrid account that got compromised, after which all of our comms were (predictably) being marked as spam. After back and forth with support, we were told to continue sending as normal and that "eventually" our genuine emails would stop getting sent to spam.
That lacklustre support response was enough to put me off the service. And then, funnily enough, a few days ago I received a reply from Sendgrid support about another account I had opened for a side project. This reply came in ~2 months late, where I was asking if I could be moved to a different IP range as all my emails were getting blocked by Outlook services due to the shared IP I was assigned being (seemingly) misused by spammers. Again, the response was "just keep trying as normal" and eventually my emails would work... Apparently. That's obviously not a viable business communications strategy.
The funniest part about both these support responses was that despite one of them taking several weeks to get a response, I'm being pinged every few days asking me if my issue was solved. It's all well and good for me to wait months when I need a response as to why all my emails are getting sent to spam, but as soon as the ball is in their court, they don't mind pestering me for a response.
I don't have enough bandwidth for a dedicated IP, so I signed up for a service that requires a credit card even for its free plan, which I believe is a decent enough barrier against spamming.
That lacklustre support response was enough to put me off the service. And then, funnily enough, a few days ago I received a reply from Sendgrid support about another account I had opened for a side project. This reply came in ~2 months late, where I was asking if I could be moved to a different IP range as all my emails were getting blocked by Outlook services due to the shared IP I was assigned being (seemingly) misused by spammers. Again, the response was "just keep trying as normal" and eventually my emails would work... Apparently. That's obviously not a viable business communications strategy.
The funniest part about both these support responses was that despite one of them taking several weeks to get a response, I'm being pinged every few days asking me if my issue was solved. It's all well and good for me to wait months when I need a response as to why all my emails are getting sent to spam, but as soon as the ball is in their court, they don't mind pestering me for a response.
I don't have enough bandwidth for a dedicated IP, so I signed up for a service that requires a credit card even for its free plan, which I believe is a decent enough barrier against spamming.
> That lacklustre support response
Frankly, this makes my blood boil. Full disclosure: I used to work at Sendgrid's competitor (Mailgun) and I'm quite familiar with what's happening under the hood. You allowed spam to be sent from your account. You poisoned the IP space (not just one address you were sending from, but the IP block, affecting others) with spammy reputation. These blocks are expensive and hard to acquire (due to IPv4 shortage) and at this point you have caused more damage to Sendgrid and their customers, than your account value can probably compensate for. What kind of "support" do you expect at this point? You rented a hotel room, gave the key to someone who erected a meth lab there, and you're unhappy with the hotel's reaction?
Sending email on behalf of other people is hard work. The margins are not great, the ratio of spam-to-ham for new accounts is insane, and they're constantly under pressure between their customers who pay very little but want to spam the entire universe (I am sorry but most of you do!) and ESPs not wanting that traffic.
Frankly, this makes my blood boil. Full disclosure: I used to work at Sendgrid's competitor (Mailgun) and I'm quite familiar with what's happening under the hood. You allowed spam to be sent from your account. You poisoned the IP space (not just one address you were sending from, but the IP block, affecting others) with spammy reputation. These blocks are expensive and hard to acquire (due to IPv4 shortage) and at this point you have caused more damage to Sendgrid and their customers, than your account value can probably compensate for. What kind of "support" do you expect at this point? You rented a hotel room, gave the key to someone who erected a meth lab there, and you're unhappy with the hotel's reaction?
Sending email on behalf of other people is hard work. The margins are not great, the ratio of spam-to-ham for new accounts is insane, and they're constantly under pressure between their customers who pay very little but want to spam the entire universe (I am sorry but most of you do!) and ESPs not wanting that traffic.
I don't think I did a great job of making clear that my main gripe with their support (taking months) was from a separate, personal account which — from day one — was unable to send non-spam, transactional email to anyone with an Outlook address. In my first example, the account's (for previous employer — not "me") right to support I guess is more debatable...
But I don't think your analogy of the hotel room is totally representative here. Without knowing how someone has had their credentials hacked, it's much more prudent to assume that in your analogy that the room key was pickpocketed / stolen. And then it becomes more of a grey-area as to how much support one can expect.
That being said, I do appreciate your insight on account value, as compromised accounts clearly do constitute a burden that don't end up paying for themselves (even if they aren't "to blame").
But I don't think your analogy of the hotel room is totally representative here. Without knowing how someone has had their credentials hacked, it's much more prudent to assume that in your analogy that the room key was pickpocketed / stolen. And then it becomes more of a grey-area as to how much support one can expect.
That being said, I do appreciate your insight on account value, as compromised accounts clearly do constitute a burden that don't end up paying for themselves (even if they aren't "to blame").
> And then it becomes more of a grey-area as to how much support one can expect.
Lol no it doesn't. The comment you replies to uses an excellent example, this isn't someone snuck into your room with the stolen key and messed it up, they burned the room down.
Their support isn't coming after you for the value of the room, but they're also politely telling you they aren't about to replace your belongings or give you a new one
Lol no it doesn't. The comment you replies to uses an excellent example, this isn't someone snuck into your room with the stolen key and messed it up, they burned the room down.
Their support isn't coming after you for the value of the room, but they're also politely telling you they aren't about to replace your belongings or give you a new one
But notice how that doesn't happen very often. Because hotel keys almost never have the room numbers on them. This is basic security the hotel provides to handle the inevitable fact someone is going to lose their credentials. The Sendgrid equivalent would be some mechanism to prevent lost credentials leading to abuses - such as 2FA.
I'm actually intrigued now that I think about this.
What do you think typical hotel keys actually are? They could be arbitrary one shot token random tokens authorised for your stay. When you check out your token, even if you've cloned it, is now useless. This would superficially match the UX you see used, in which each mag stripe card is rewritten before it's handed to you when you check in.
But from what I can tell actually the typical practice is that the card doesn't have a random token, it encodes the room number and period of stay. If you write a new card with a different room number and period it would work, although of course that doesn't make such a thing legal to do.
I think the lack of a human readable number on your typical hotel keycard is because it was easier/ cheaper not because of some security insight. I would be happy to be proved wrong.
Certainly when I've stayed at very small hotels with actual keys, the keys were marked with a room number. These hotels also really wanted the keys back when you check out of course, not because they think you'll come back later and enter a room that's now empty or has a different guest (at such a small hotel that would not be subtle) but because they need it for the next guest.
Anyway. Sendgrid's 2FA doesn't actually block lost credentials. If you have Sendgrid 2FA and use it to get a token for their API, and then the new guy puts it on Pastebin your token will now be abused to send spam.
The main benefit is that the random tokens aren't guessable whereas your brilliant choice of Sendgrid password, "sendgrid" is very guessable. Yes this is some very weak sauce.
What do you think typical hotel keys actually are? They could be arbitrary one shot token random tokens authorised for your stay. When you check out your token, even if you've cloned it, is now useless. This would superficially match the UX you see used, in which each mag stripe card is rewritten before it's handed to you when you check in.
But from what I can tell actually the typical practice is that the card doesn't have a random token, it encodes the room number and period of stay. If you write a new card with a different room number and period it would work, although of course that doesn't make such a thing legal to do.
I think the lack of a human readable number on your typical hotel keycard is because it was easier/ cheaper not because of some security insight. I would be happy to be proved wrong.
Certainly when I've stayed at very small hotels with actual keys, the keys were marked with a room number. These hotels also really wanted the keys back when you check out of course, not because they think you'll come back later and enter a room that's now empty or has a different guest (at such a small hotel that would not be subtle) but because they need it for the next guest.
Anyway. Sendgrid's 2FA doesn't actually block lost credentials. If you have Sendgrid 2FA and use it to get a token for their API, and then the new guy puts it on Pastebin your token will now be abused to send spam.
The main benefit is that the random tokens aren't guessable whereas your brilliant choice of Sendgrid password, "sendgrid" is very guessable. Yes this is some very weak sauce.
Lol what?
Maybe the analogy is getting a little long in the tooth here, but it's been 4 years since you needed basic auth on Sendgrid, so hopefully your username and password can't be found together without a targeted attack (kind of like how your room key getting stolen should only lead to your room with a targeted attack).
On the other hand, many hotels will give you a room number on the sleeve of the card. It's up to you to do the right thing and get rid of it properly
Kind of like Sendgrid has 2FA and it's up to you to set it up.
I mean, I get it, default behaviors don't rely on users doing the right thing, phone 2FA doesn't count even though it would have probably saved OP just fine, etc. If Sendgrid was trying to come after OP for losing their credentials those would all be big factors in me wanting to boycott Sendgrid...
But they're not. They're pretty much just telling OP "it is what it is after you let your account get stolen, eventually you might recover from this but you don't get a free second chance"
Maybe the analogy is getting a little long in the tooth here, but it's been 4 years since you needed basic auth on Sendgrid, so hopefully your username and password can't be found together without a targeted attack (kind of like how your room key getting stolen should only lead to your room with a targeted attack).
On the other hand, many hotels will give you a room number on the sleeve of the card. It's up to you to do the right thing and get rid of it properly
Kind of like Sendgrid has 2FA and it's up to you to set it up.
I mean, I get it, default behaviors don't rely on users doing the right thing, phone 2FA doesn't count even though it would have probably saved OP just fine, etc. If Sendgrid was trying to come after OP for losing their credentials those would all be big factors in me wanting to boycott Sendgrid...
But they're not. They're pretty much just telling OP "it is what it is after you let your account get stolen, eventually you might recover from this but you don't get a free second chance"
I really don’t understand this - I’ve read the parent comment 3 times and can’t see how you you come to the “gave someone the key to someone else” analogy.
Are we interpreting OP differently? From all I can read of available information, they seem to have been reasonable. Sounds more like their neighbor or the previous tenant had a meth lab.
Are we interpreting OP differently? From all I can read of available information, they seem to have been reasonable. Sounds more like their neighbor or the previous tenant had a meth lab.
> At work we had a Sendgrid account that got compromised
Parent commenter's account was compromised.
Are you assuming its SendGrid's fault the account was compromised? I'd assume the opposite.
Parent commenter's account was compromised.
Are you assuming its SendGrid's fault the account was compromised? I'd assume the opposite.
Especially given that they haven't had working 2FA for the longest time, it's not fair to say that OP allowed it or gave someone the keys.
Stretching the analogy, it's more like OP had the room keys lying on a table at a cafe, someone managed to copy the key after sniping a photo and then abused the room while OP was out for the night, and then OP notified the hotel staff once they got back and realized what was going on.
In both cases, OP was unaware of anyone having unauthorized access and it wouldn't have happened (as easily( if the business had better security.
Stretching the analogy, it's more like OP had the room keys lying on a table at a cafe, someone managed to copy the key after sniping a photo and then abused the room while OP was out for the night, and then OP notified the hotel staff once they got back and realized what was going on.
In both cases, OP was unaware of anyone having unauthorized access and it wouldn't have happened (as easily( if the business had better security.
I was sure this was satire of the kind of response you'd get from any run of the mill mail provider. I can't believe you're being serious...
As a sendgrid customer, this explains why, even after upgrading to dedicated ips and setting all our auth/security stiff correctly we still have emails being blocked inexplicably. Top off the rubbish customer service - they didn’t respond to me about getting blocked by outlook on our private IP until I tweeted them... - and next week looks like my job is to move to postmark.
Our customers are also getting our emails flagged as malware. Great way to onboard a service and verify your email, with a Google Malware alert. :/
Its Google's world and Twilio is living in it. :|
Its Google's world and Twilio is living in it. :|
Sendgrid have a HUGE problem with spammers using their service. I've probably reported 20 SPAM campaigns using their official channels, but I think they route such e-mails to /dev/null. Never gotten a response- just more SPAM through their service. And I can't block their IPs, as a lot of legitimate sites utilize their services.
Related, 2 months ago my sendgrid account gets hacked and a stolen (I presume) credit card gets added. They buy a plan, then proceed to send spam/phishing. I of course contact their support within minutes of receiving an email that $750 was spent on my account. They've now spent $1500 through my account on these spam emails, yet sendgrid finally dealt with the issue yesterday by deleting my account.
https://mobile.twitter.com/PaulBGD/status/129591880033708441...
https://mobile.twitter.com/PaulBGD/status/129591880033708441...
This article sounds like sendgrid is the victim. But in reality this is going on since months. My account was hacked three months ago and abused by spammers. There was no way to stop it in the UI and the support didn't react for days.
In the end I had to pay ~1k$.
I guess Twilio will close down sendgrid soon anyway and this is a way to monetize the last few months.
Stay away from Sendgrid and Twilio, they are incompetent and/or aholes.
I guess Twilio will close down sendgrid soon anyway and this is a way to monetize the last few months.
Stay away from Sendgrid and Twilio, they are incompetent and/or aholes.
Yep, this happened to our ORG a while back, and Sendgrid took no responsibility. Wish I could go into details, but... they're finally implementing 2FA and revamping how they deal with identities.
It's similar to the github pages exploit from years ago, which is why we now have github.com and gitub.io
The purpose of these mails is of course to get credentials and spam some more.
At first I checked for a bounty, but as this is a "configuration error" (SPF), there is no bounty.
Then I sent it by mail, but yeah.. No reply.