> The screenshot I got of your conversation with (Dave) was, indeed very well done and was both informative and actionable. You responded to us very quickly and with a high level of professionalism.
> Further, participating here in this HN discussion is noteworthy and impressive.
Thank you - much appreciated.
The quality, morals and perception of our product and how we treat people matters to me greatly which is why I'm always happy to get involved in discussions like this.
As is evidenced from other post on this topic, there are a number of competitors that don't behave in the same way and that leads people to think we are all the same and I absolutely want to challenge that perception when it comes to us.
> FWIW, we immediately complained to he.net about the bad actor elsewhere in that /24 ... we'll see.
Thanks, feel free to refer them to me and I'm happy to provide as much evidence as they need.
> Bottom line: our duty of safeguarding customer data trumps this weeks fashionable spam heuristics.
I haven't suggested that you do anything differently to that.
Keep the data, keep the accounts, do whatever you feel is best for you and your business.
All I've said is to be smarter when it comes to sending email to old accounts that are repeatedly bouncing. I've outlined what was wrong and I worked with your engineers to resolve it.
From our perspective, we don't link IP and domains at all at the moment, they are totally independent.
Because spam traps get very different traffic to a real mail server, you can make a lot of assumptions that would be impossible to do on a real mail stream.
So generally speaking, if your domain name isn't in the top million domains (and there are still some exceptions to this - but I'm not going to list them all here) and you hit one of our pristine traps (see my post about the different trap types) with a message containing your domain, that will usually cause the domain to be listed.
Other lists and spam filters will have their own methods for this.
My belief is that Google and Microsoft only use AI for this (which is a bit of a nightmare if it gets it wrong).
Sure - happy to elaborate here, re-reading my response I understand the confusion here as my choice language wasn't great ;-)
I didn't say anywhere that you MUST support HTTP unsubscribes, but I just said it was preferable (to me at least).
HTTP typically means there is some sort of form which will typically auto-fill the email address from the database (because of the HTTP GET parameters included in the link) and all the user has to do is generally click 'Unsubscribe' and the address is automatically removed from the database.
The reason I prefer this is that if I or one of my colleagues want to unsubscribe a spam trap, then we can - but only if the above is true. (Disclaimer: we only do this very occasionally, usually it's only if we've spoken to someone and are convinced they are legitimate, have fixed any issues we've identified, but are hitting traps because of a single address - if we didn't do this then we'd either have to force the sender to reconfirm their entire database by asking everyone to opt-in again, or we'd have to whitelist the sender, neither are great options - so if we ever do this, then it's done hours/days after the event so as to not tip off anyone on what the trap was).
Because spam traps have to stay secret, we only reference them by a hash value internally (and only I and a select few can convert hash to trap value to do the unsubscribe) and as the messages are received on our trap network the traps themselves are obfuscated to a know value before they are stored in our evidence system.
mailto: means we can't do this at all - a trap can never originate traffic, that's cardinal rule of them, it would be grossly unfair if they did.
> - What do you mean by unsubscribe without contacting me? Isn't HTTP or SMTP just as much contacting me?
Poor use of words from me.
What I meant is that I don't trust messages that have an unsubscribe that does not encode the recipient in it in some way e.g. HTTP method without GET parameters (as this means I would have to do considerably more work to fill out the form) or a mailto:[email protected]?subject=unsubscribe - both to me scream our that there is no automatic connection to a database and that someone on the receiving end has to manually remove the address - which could take weeks, or be never.
One of the ways most blocklists work is by employing spam traps which can either be individual email addresses or entire domains.
It's quite a large topic, but I'll try and summarise for you.
You can't just take a spam trap and use it to block anything that hits it - that would be incredibly unfair as you might not know the history of the email address or domain and you'd generate considerable false positives if you did.
If you buy a domain or register an email address and you know for certain that it's never been used before, then this is typically called a "pristine trap" and there are then various ways to then "seed" this so it starts to receive spam and malicious traffic. This is the exception where it could be used immediately. However this process usually takes a very long time (usually years) to receive enough traffic to be useful.
Any other address or domain where you don't know the history would be called a "recycled" trap. There are various opinions on how these should be managed, but generally it's accepted that you need to reject ALL traffic on these for at least 2 years to allow genuine senders to work out they are no longer valid (which is why bounce handling is important!).
The other type of trap is a "typo" trap, which is typically an entire domain (or an email address deliberately close to another) which could easily be typo'd. I have a rule here that says we neverever use these for any blocklisting (other lists will not have the same rules!), however they can be useful to see trends and detect compromised hosts (as these typos frequently end up in compromised databases that get sold around the dark web). They can receive considerable volumes as well.
Typo traps are especially why you should always use confirmed opt-in and employ CAPTCHA on your sign-up forms (although this also helps with all traps in general).
We carefully monitor all traps to see how they are performing e.g. detections .vs. traffic from whitelisted sources and false positive reports and will immediately remove a trap if it begins to perform poorly or has been made public in any way.
It takes years to build a trap network and we have to discard all of that work should they become discovered (as they could be used maliciously), which is why a blocklist operator will never reveal the addresses to you as they are a closely guarded secret. In fact on our own network, even we don't see them - we monitor them via a hashed name! (only I and a few other select people can convert the hashes to the actual domains or addresses).
I think that covers the basics - I hope that explains it sufficiently for you
Apologies for not being clearer on this, I'm generally talking about marketing messages when I say "not keeping in touch".
My point here is "consent to send", e.g. I think most people would find that someone importing a years old list of email addresses and then suddenly sending marketing messages is unacceptable and constitutes spam.
If you're "doing it right", then "touching" your customers with a once a year "Hey, we haven't seen you in a while" message is absolutely fine and generally a good thing to do as it maintains consent, because they can unsubscribe if they're no longer interested (as they might have forgotten they signed-up or had an account).
If someone hasn't logged into your "service" for >1 year and now wants to do a password reset several years later, then I would say that is entirely different.
We generally try our hardest to handle this case without it causing a listing if you hit our traps with something like this.
I'm the Abusix engineer in question (and actually the architect of the system in question), and you're being somewhat "economical" with what actually happened here.
Here's the actual chain of events in question:
- You recently switched ISPs and that meant you moved to a new IP block.
- Without going into massive details on how our infra works, but when we have the most serious level of listing (e.g. hitting our most secure traps as in this case), we treat the same /24 more aggressively than we normally would (because of things like snoeshoe spam that would normally spread traffic across a wide range of IPs).
We use /24s only where we cannot determine different ownership of the IPs by looking at the abuse contact registered at the RIR e.g. if the IPs are different contacts then we don't bundle them into the same bucket. In this case Hurricane Internet owns the entire range, so the /24 is used.
We do this because there is no other way to do it that isn't completely abuseable by a bad actor e.g. rDNS is completely trivial to forge and to claim multiple fake entities. If someone has a fool-proof way to do this, then I'm all ears.
Then we get to your failings in this case:
- You don't have proper, working bounce handling.
You were repeatedly sending mail to an old customer and we were rejecting 100% of the email you sent to that address - stating that you should stop sending to it in the rejection message. We always reject traffic on traps we are building so that bounce handling removes them automatically over time.
- You decided to send a marketing message to a bunch of users, including to the address described above "We (rsync.net) are experimenting with a lifetime prepayment option". This message provided no List-Unsubscribe option at all, so I could not unsubscribe it without exposing the trap to your engineer (which is my primary concern as it takes years to build traps properly).
- Your engineer said to me "we took down 600 old accounts and are reviewing our contact policies going forward" and "there are still plenty of other customers who are listed generically as bouncing so we will have work to do here".
That tells me that you knew that you were sending to old accounts and that your bounce handling was either not great or non-existent.
I take our role very seriously and I and my team go out of our way to help anyone who finds themselves listed by providing evidence and advice and we always try to find a good resolution for all legitimate senders.
That is exactly what I did here with your engineer, the problem is resolved as the specific account was removed and you know now what you were doing wrong and how to fix it so that it never happens again.
Your engineer was appreciative and I said if there are any further issues in the meantime whilst you fix things on your end, then we would help by exempting your traffic whilst you did those changes.
I can't see how we could have done anything more in this case.
On our website, we provide blog posts and videos, we take part in conferences and workshops to give advice on how to do things properly so you never have blocklisting issues.
I can easily summarise here what everyone should do to avoid issues:
- Have proper rDNS for your sending IPs that is part of your administrative domain (e.g. don't use your providers generic rDNS that contains the entire IP in the hostname). If someone visits the domain name, make sure there is a website present that has contact details as a minimum.
- Make sure your abuse@domain and postmaster@domain accounts actually go to a responsible person.
- If you send any marketing mail at all, make sure it has a working List-Unsubscribe header (preferably HTTP that allows someone to unsubscribe without having to contact you). Important note: If you use a mailto: unsubscribe, then I cannot unsubscribe a trap, even if I wanted to.
- Make sure you have working bounce handling. If you repeatedly send to an account and it either bounces or is hard rejected, then you need to stop sending to it and mark it as bad. No excuses.
- Don't send to addresses where you have not contacted them or had any interaction at all for > 1 year. If you haven't kept in touch with your customers, then that is on you.
- If you have a web form that when submitted, sends a message to an external user, then you MUST a) validate all of the input fields and disallow URLs unless the field required it and b) prevent automated submission via the use of a CAPTCHA.
- When collecting email addresses for mailing lists, always use confirm opt-in e.g. send a message containing a link that they have to click to activate the subscription. Do not send any further messages to them until this has been completed.
- Make sure you separate IP addresses being used for outbound mail from those used for outbound NAT pool. Block outbound port 25 from the NAT pools and make your firewall notify you of any port 25 activity from any hosts as this could indicate they are infected.
- When provisioning a new mail server IP, don't send more than 30 messages per minute (e.g. 0.5 messages/sec) for the first day and then increase the volume over the following week.
I'm sure some will disagree with some of these, but I guarantee that if you follow all of these, then you'll never have an issue with a blocklist like ours.
Not all blocklist providers are the same.