return to table of content

A deep dive into email deliverability in 2024

xyst
42 replies
16h56m

Given how much weight “Gmail”, “Outlook”, and “Yahoo” email providers pull, I have always wondered about a different type of attack on business entities: “targeted failed deliverability”

Basically in this attack, a victim (particularly a business or mailing list or NGO) is sending out bulk emails to which the attacker owns. Even sourcing this out to shady off shore click farms would work too.

Attacker then marks the victim’s emails as spam in Gmail/Yahoo/Outlook. The “AI spam filters” pick up on this new “spam activity” and will then mark future emails as spam or even delete them before reaching real customers.

After a year, company bleeds money on a quarterly basis. Ad departments wonder why there is decreased engagement through email. Technical departments are bamboozled.

Maybe a big company will be able to weather the storm or just ditch email altogether. But small companies would definitely take a hit. Even smaller NGO or political mailing lists would lose donations (assuming email was a significant source of new donations).

Probably a very low vector of attack tbh, but something that has lingered in my mind.

nemomarx
14 replies
16h53m

A webforum I know has a rule against marking email notifications they send at spam (You can opt out of receiving them through the site, they just don't want you doing it on the email client end) to avoid this happening to them. For a small org, it's kind of a real risk?

I'm not sure how larger orgs mitigate it.

uuddlrlrbaba
4 replies
15h53m

They pay 3rd parties to handle relay duties. A large chunk of reliable delivery is based on those received headers, aka what systems your mail is relayed through.

jimkoen
3 replies
13h43m

Lol wait, that sounds like e-mail deliverability is almost set up like cartels.

77pt77
1 replies
12h47m

Mass sending is controlled by sendgrid and the likes of that.

ozr
0 replies
12h48m

Cartel is maybe overly pejorative, but it's definitely based on relationships.

Postmasters at large ESPs and inbox providers can and will text each other to resolve issues.

This is pretty much how the internet works as well (BGP, etc). It's opaque, but open.

bboygravity
4 replies
10h40m

I run a small org (freelancer) and I have a super advanced bullet proof set of solutions against this. Any 1 of those should do the trick (I do all of these):

1. Don't trick people into signing up for mailinglists.

2. Don't spam.

3. Don't use mailinglists.

My small business is fine.

spacebanana7
3 replies
9h48m

Not against a targeted attack like the one described.

A clickfarm marking password reset emails as spam could create real harm.

Even a malicious individual doing this to 2-3 fair transactional emails could cause damage for a small business with low volume.

jeromegv
2 replies
5h45m

The attack described could not work with a strict DMARC policy.

However of course, most small companies have no idea how to do SPF/DKIM/DMARC.

spacebanana7
0 replies
5h33m

Could you explain how DMARC would prevent this?

If I attempted to create dozens of accounts on a website, trigger password resets emails, and report them as spam, where would DMARC prevent me?

pc86
0 replies
4h8m

Nobody is spoofing anything, DMARC won't help. This is marking legitimate emails as spam at scale to trigger actual spam controls against the legitimate organization.

KTibow
3 replies
14h10m

Is there any way for them to enforce that

buro9
2 replies
9h26m

I run webforums, and this is enforced trivially.

I don't send marketing, I only send transactional emails. Notifications are opt-in, and transactional. Logins / signups are done via sending a TOTP to the email, these too are transactional.

If someone marks notifications of new posts in a subscribed thread as spam... fine, but this is self-solving, as this person will trigger no more email ever reaching them for that email address, meaning they also cannot now sign in to the website, and therefore cannot subscribe to more email updates, or visit things to refresh "last viewed", and so would never be notified again.

Emailing a TOTP as a login has increased deliverability by self-selecting the removal of those who hit the spam button. Deliverability of email from the webforums is over 99%.

Reporting email as spam, effectively bans oneself from the website. I didn't even need to do anything.

saurik
1 replies
8h36m

When you report something as spam to Google they only claim they "might" cause later emails from the same sender to go to your spam folder; regardless, those emails going to one's spam folder doesn't mean they don't get them at all, it just means they have to look at their spam folder to log in.

buro9
0 replies
8h29m

When you use services like Sendgrid, having a spam report will automatically prevent future emails being deliverable, third parties and intermediaries do this precisely to protect their email reputation.

Marking spam may not be immediately be a death knell within Gmail, Hotmail, etc, but because of the potential impact from being marked as spam, virtually everything treats it like a death knell.

This is fine for me, my service is entirely opt-in, and if someone hits the spam button it risks impacting other users who _want_ the email, so I am not bothered that this person effectively unsubbed themselves and killed their account.

elorant
5 replies
7h20m

That’s why most e-commerce sites send their marketing stuff from a different domain. If it gets flagged they can still send transactional e-mails from their main domain. Assuming that both mail servers are on different ips.

inopinatus
3 replies
6h42m

This seems kinda deluded given that spam prevention teams have been identifying domain clusters for sender reputation management since at least 1998 to my earliest direct knowledge and probably earlier. Maybe it works for a little while, but don't bet your company on it.

elorant
2 replies
5h49m

That's not spam though. It's promotional material sent to subscribed users.

hiddencost
1 replies
5h37m

This is dishonest.

elorant
0 replies
5h22m

Why? You send to subscribed users with links to your main site. There's nothing fishy in the concept. You're not trying to fool anyone.

Natfan
0 replies
1h52m

I'd argue transactional email should also be sent on a subdomain, to allow for a migration if that subdomain gets "burnt". I'd also argue that really nothing should send email from your root domain at all for this exact reason, but legacy environments can often make this a non-starter.

semanticist
3 replies
8h2m

I don't think this would work in practice. My employer sends daily deal emails without using a third-party service like SendGrid or SES and what we do is pay a company like Validity who interface with the big email providers for us. They have honeypot emails that get and validate our emails, they get feedback from the providers on how much they like/dislike us, and we get reports on this.

So an attack like this would be very obvious very quickly, even leaving aside that we'd notice a huge spike in email sign-ups and probably kill their accounts (especially since they're not going to be buying anything from those sock puppet accounts!).

altdataseller
1 replies
7h38m

You’re assuming most small companies are as vigilant as your company

tootie
0 replies
5h40m

If email is business critical, you do it. I work for a pretty small company and we do stuff like this. We have a sender tool that gives us a static sender IP, reports deliverability, click rates and at least estimates open rates. We also have a tool to estimate the quality of a newsletter sign up email address and not collect any disposable emails.

lippihom
0 replies
7h28m

If they're not using a third-party service what are they using? Daily deal sounds like it's very high volume...

kirse
3 replies
11h14m

Yea I've thought about this but not from the "attack on entities" angle but moreso a consumer-rights / boycott angle. I've had a negative enough experience with a large "maximizing shareholder value" company that I went back through my email history and marked every single one of their comms as spam.

Might be a drop in the bucket, but it doesn't take many votes to make a difference in the spam world.

I'm sure this will evolve soon enough and email delivery might increasingly become pay-to-play with all sort of backroom agreements, if it isn't already.

janalsncm
2 replies
10h47m

I bought my dad a sweater for our local MLB team. I made the mistake of using my real email. Ever since I’ve gotten a steady drumbeat of marketing emails and other low value content from them.

Spammers want us to think there’s a significant difference between their newsletter or marketing notes we may have technically signed up for (certainly not willingly) and I don’t feel bad about reporting both of them. If this forces spammers to consider whether recipients will want their messages, good.

pc86
1 replies
4h11m

It sounds like you absolutely did sign up for the emails, though.

I'm not sure how you could "unwillingly technically sign up" for something like that, especially at the scope of an MLB team which is going to have a team of lawyers, marketing policies, etc. They're not just going to spam people the risk is way too high.

inetknght
0 replies
3h34m

It sounds like you absolutely did sign up for the emails, though.

Did he? The anecdote here is probably observed by everyone on this forum. How odd that you find it's unlikely to receive spam from a business transaction.

I'm not sure how you could "unwillingly technically sign up" for something like that

Have you tried using the internet?

the scope of an MLB team which is going to have a team of lawyers, marketing policies, etc. They're not just going to spam people the risk is way too high.

I would love to live in your world where there's little likelihood of getting spammed just for purchasing something once. Unfortunately, spamming people has effectively zero risk and all reward. If there were any real risk then we would see actual real and frequent consequences every day. We don't see that, but we do see lots of spam in our inboxes.

BeFlatXIII
3 replies
5h36m

political mailing lists would lose donations

Now you have me rooting for the bad guys.

chgs
2 replies
5h11m

If political candidates can’t get small donations then that shifts more power to large donations 1 wealthy companies, people, and unions (maybe not the last in the US)

pc86
0 replies
4h4m

Unions in the US donate hundreds of millions of dollars a year to political candidates.

flkiwi
0 replies
1h24m

You know what drives me not to be involved? Knowing that buying a mug from a candidate I like virtually guarantees a torrent of emails from every downticket race in markets I’ve never lived in. And because of the way they set up the lists, unsubscribing means unsubscribing only from “Kelleher for Coroner” in a county halfway across the country. The worst part is the incredible entitlement you experience if you mention this to someone involved in a campaign. I continue to vote, but I haven’t donated a penny to a campaign in more than a decade, specifically to avoid the harassment.

Edit: USian speaking

jeromegv
2 replies
5h48m

Isn't that what DMARC policy would prevent? If the emails being sent by the attacker are failing SPF/DKIM, then we can configure the DMARC policy so that Gmail never delivers those fake emails in the first place. So that attack would not be happening.

bennettnate5
1 replies
4h27m

The attacker isn't sending mail spoofed from the domain--they're intentionally signing up for legitimate newsletters from that domain using a ton of email accounts they control, then reporting those newsletters as spam in each email account.

pc86
0 replies
4h9m

This could be a pretty interesting use case for a botnet as well, compromise computers, compromise email accounts, for the sole purpose of selling the ability to mass-spam your competitors. Obviously immoral but simply from a technical standpoint I'm wondering if it would work and what scale you'd need for it to be effective.

pompino
1 replies
10h54m

This is Google's business model, they throw completely legitimate emails your business sends into spam/marketing, so you're forced to pay them for gmail ads.

PawgerZ
0 replies
4h36m

If your email can be displayed in a gmail ad, then it is spam/marketing.

dns_snek
0 replies
42m

Do political mailing lists get through in the first place? I don't live in the US, but someone must've sold one of my throwaway gmail addresses thinking I did.

Every couple of months I end up checking its spam folder and it's just a daily barrage of spam from both DNC and RNC, 1-2 emails per day like clockwork. None of them ever got through to the inbox though.

bongodongobob
0 replies
14h14m

That's just email working as intended. You pay a third party a small monthly fee to handle email blasts via a relay.

Neil44
0 replies
9h24m

You don't typically use your main email system for bulk sending, you use a third party for that who is used to taking that heat.

Fatnino
0 replies
45m

I know of victims who had their legit email template lifted by actual slammers. The spammers would embed the legit template invisibly in their emails and then only have a few short lines visible with the actual scam. The idea being that filters would see the majority of the email is legit looking and let it in. Eventually users would flag enough of these as spam and the template itself would trigger the blocks. Then the spammers would move on to the next victim who's email template still gets through filters.

Meanwhile the first victim is left to pick up the mess where none of their email gets through anywhere.

ttul
15 replies
19h11m

This change was necessary and long overdue. Requiring domain owners who send significant volumes of email to properly sign their messages allows receivers to more clearly delineate good from bad based on domain reputation rather than IP address reputation.

As more domains send email through shared IP space on transactional and marketing services, having the ability to attach reputation reliably to the sender domain is incredibly helpful in reducing abuse.

adrian_b
9 replies
18h16m

The condition about "significant volumes" is not true.

Google states that the new requirements are mandatory only when you send at least 5000 messages per day.

This is a lie. I send at most a few messages per day and usually less than one per day was towards a gmail account and I had implemented a part of the requirements, but not all of them.

Nevertheless, Google has started to reject my messages, so I was forced to waste time with the implementation of all requirements, even if they are somewhat redundant.

denton-scratch
3 replies
10h26m

My domain also sends no more than 10 messages a day. The domain is correctly-configured with SPF, DKIM and DMARC.

At the beginning of March, I started getting temporary rejections from gmail. Not all of my outgoing messages, maybe 1 in 10. Most of these messages were to one individual, to whom I've been sending for years. My domain has existed continously since 2002, and has never sent spam. The rejection message was startling: words to the effect of "Your message has been rejected because of the awful reputation of the sending domain". The reputation of my domain is spotless, according to various testing tools.

There have been no new rejections in the last two weeks.

According to TFA, Google started rejecting a proportion of mail from bulk senders in February. I wonder if I got caught up in some half-baked roll-out of this new (old) policy.

jeromegv
1 replies
5h44m

Did you link your domain to Google Webmaster tools? It will let you know what your domain reputation is.

denton-scratch
0 replies
5h9m

No, I did not. I barely ever use Google services of any kind.

My "domain reputation" should mean something like "the consensus of multiple reputation services", where those reputation services are reasonably open about how they score. It shouldn't mean "the completely opaque opinion of a single, hyper-dominant, secretive provider with all kinds of conflicts of interest".

drdebug
0 replies
9h8m

Same here with 20+ years old mail service on the same domain that has never sent spam with correctly configured DNS SPF DKIM DMARC, getting gmail rejections. I noticed a significant improvement after linking the domain to my google account https://support.google.com/mail/answer/9981691

skrause
2 replies
17h38m

Maybe someone else sent 5000 mails to Gmail users using your unauthenticated domain.

adrian_b
0 replies
10h45m

That would not have been possible, with the already existing checks.

While the individual messages were not yet signed, Gmail should have already rejected any message claiming to be from my domain that was not sent by my own server.

sdfhbdf
0 replies
2h20m

It also says that you only need to hit this limit once and you’re forever on that list. Also domain is considered as the canonical domain, the part right before the TLD. Subdomains are counted together with the apex domain.

janosdebugs
0 replies
6h13m

I can confirm this, I have a mail server with extremely low, non-bulk mail traffic and without SPF/DKIM/DMARC nothing works. Even with, Gmail started rejecting mail (probably noisy neighbor), so the only way around that was to use the little-advertized relay service in the paid Google Workspace when sending towards gmail or workspace domains.

cj
2 replies
18h13m

As a sys admin, what do you do when you see 5% of your email hitting spam because the recipient’s Office365 mail server is misconfigured?

Agreed it’s a net positive, but it kills me when the reason emails land in spam is misconfiguration at the recipient’s end. (Like forwarding emails which breaks SPF)

brightball
0 replies
17h57m

As long as DKIM is configured correctly, it shouldn't matter.

BeFlatXIII
0 replies
5h31m

My employer has had massive issues with this recently. Lots of office chatter from HR about needing to call candidates b/c the response rate is down this year due to spam folders eating the offers.

xoneill
0 replies
19h2m

Agreed, this is the upside for sure. Despite frustrations, I'm hoping this cleans things up.

ozim
0 replies
5h55m

We also slowly but surely are moving towards IPv6 which will make reputation based on IP somewhat useless when I can have as many new IP addresses as I want. They would have to make all newly seen IPv6 addresses not trusted by default when bad actor could send each email from different IPv6 address.

77pt77
12 replies
19h11m

My personal VM has just been placed in some RBL because the entire /24 address space was blacklisted.

Someone (allegedly) sent SPAM and now my machine that sends maybe 3 emails a week is blacklisted

xoneill
9 replies
19h4m

That sucks! And fixing reputation is at best a nightmare. I've seen suggestions about purchasing / sending email from other domains - to protect your primary domain. Not something I really care to do.

77pt77
4 replies
15h49m

The "de facto" solution is to outsource that to Google.

Email on a personal machine and domain has been dead for over 10 years.

You just can't own your data.

You can receive it, no problem. But you can't send it.

akira2501
2 replies
15h10m

The "de facto" solution is o outsource that to google.

So, they have a "de facto" monopoly, or at best, are working with other providers to create a cartel.

Email on a personal machine and domain has been dead for over 10 years.

Due to the actions of?

You can receive it, no problem. But you can't send it.

You can send it. They're actively deciding to just block you. Then provide you no recourse.

77pt77
1 replies
15h6m

The monopoly came about "organically". It's not like there was a conspiracy.

You can send it

Not even that. Nowadays it's not uncommon for some servers to even refuse the connection.

Mind you, I'm not talking about aggressive spammers.

And yes, there's pretty much no recourse.

akira2501
0 replies
10h29m

It's not like there was a conspiracy.

I'm not nearly as sure. The mechanics of one aren't that hard to imagine. Encourage spammers and make it cheap. Don't ever fight them at their source. Invoke mechanisms that intentionally destroy public utility in public protocols. Force everyone to rely on a small handful of "reputable" senders.

Working backwards from "who decides reputation anyways?" might make it easier to see.

dugite-code
0 replies
12h22m

Sending requires quite a large volume for the big players to allow you to play. The only viable option for small servers is to use a SMTP relay service. Amazon's was a pain to get out of the sandbox mode but has been reliable and most importantly free.

As I care more about the recieving side than sending emails this works well enough for me.

cj
2 replies
18h11m

Google themselves recommends sending different types of emails from different subdomains of the primary domain to help Google differentiate between transactional, marketing, newsletters, outbound, etc

Any half decent marketer will 100% use a different domain for outbound sales (or any use case where spam rate might be abnormally high).

xoneill
0 replies
17h57m

Thank! Will look into this and reconsider.

xarope
0 replies
16h23m

absolutely. You should/must separate transactional emails (account creations, password resets etc) from EDM (marketing emails).

nickburns
0 replies
18h33m

i've personally observed an increase in this tactic from both reputable companies' blast communications and marketeers alike.

yobbo
0 replies
6h0m

The problem is that your VM provider is (unwittingly) offering the same address space to spammers. Maybe the spammer was using random addresses in the range? Or someone was starting up misconfigured smtp servers permitting relaying.

Running an outbound smtp server on a customer ip-range is going to be problematic anyway. All such ranges can be considered suspicious since the spammers who use them don't care about their standing.

mjl-
0 replies
10h38m

yes, this is a pain. the blocklist operators either seem to be not so good at vetting abuse reports, or cause collateral damage to get network operators to take action (while i don't like getting on a blocklist, perhaps it is having a net positive effect for the wider internet?).

i do think mail servers/services are using ip-based blocklists wrong. yes, you can use it as one of many signals. give it some more weight for first-time senders. but if you've been mailing (with spf/dkim/dmarc-authenticated messages/transactions), from an ip that suddenly gets on a blocklist, the previous positive reputation should be stronger than that blocklisting, and you should be able to keep communicating with your known correspondents (until they mark your message as spam after which future deliveries can be rejected/junked). it seems those mail servers/services cheap out and apply ip blocklists early in the smtp process. good for their system load, bad for their analysis performance.

in general, it seems even bigfreemail services are bad at using existing reputation in their ham/spam decisions. i recently switched an online webservice i made (that is about sending certain notification by email) to signups via email (like how you can signup to mailing lists: by sending an email to an address). the idea: if you send my service an email, i'm in your list of known correspondents. so the confirmation reply from my mail server (spf/dkim/dmarc-aligned) should certainly be accepted by yours (you much more opt-in do you want?). i tested with some bigfreemails, and yahoo put my reply with confirmation (that even references the original message) in the junk folder. to people who think you can't compete with bigmail: the bar isn't as high as you may think.

amelius
11 replies
19h18m

Isn't there any open source project that solves the e-mail delivery problem? If not, why not? This sounds like something that can be fixed by software.

jjav
6 replies
12h20m

Isn't there any open source project that solves the e-mail delivery problem?

Aplogies if you do, but it sounds like you misunderstand how email works.

The problems are stricly in policy, not in software. So there is nothing an open source project can do. The problem is that "too big to fail" megacorporations like microsoft just randomly decide to block incoming email from most of the Internet.

If email was fully decentralized (everyone runs their own server) this centralized power could not exist and there wouldn't be any problem.

(That all said, I run my own email infrastructure since long ago and it works fine. But I know some people struggle, which is contrary to the intent of the Internet.)

remus
3 replies
11h23m

If email was fully decentralized (everyone runs their own server) this centralized power could not exist and there wouldn't be any problem.

You would have different problems though. Spam would still be hard, or arguably even harder, because you would have lots of small mail servers with no idea if they are trustworthy or not. Is this uncle Bob, who I havent spoken to for 30 years, sending me a heartfelt message about the family? Or is it a scammer trying to cream some cash out of me?

jjav
1 replies
3h50m

Spam would still be hard, or arguably even harder

Doesn't make any difference. The major sources of spam today are places like gmail and outlook servers. You have to run some message classification regardless of where it came from.

remus
0 replies
2h14m

The major sources of spam today are places like gmail and outlook servers.

By volume perhaps, but is that true as a proportion of all mail that was sent via these services? I'd be surprised, though I'd be very interested if there where any numbers publicly available.

ndriscoll
0 replies
2h58m

The spam situation could also be better if we had designed into that space. Want to send a single email to a sender that you don't have an existing relationship with? Hashcash. Have a web service that sends lots of emails? Get user consent beforehand with e.g. an oauth token to bypass or lower hashcash difficulty.

arccy
1 replies
11h32m

the centralized power doesn't exist, but then you run into every misconfiguration possible. is it better? maybe.

jjav
0 replies
3h40m

Yes, it is better. The reason is that when every email operator is small, the overall email community can pressure them to go fix things. You start by sending notes to postmaster@... with problem reports. If things go ignored you start blocking them and then they wake up fast.

In today's world when microsoft or gmail or yahoo start misbehaving in email handling, what are you going to do? It is impossible to reach them and while you could block them, they will never notice and you're only hurting yourself.

It is the classic problem of centralized power which hurts everyone except the (near-)monopolists.

xoneill
0 replies
19h9m

There's some online tools out there, and an OSS project that'll scan DKIM reports for you - but this was all I could find. Beyond this, there's commercial tooling / services. I may implement something that keeps a tab on things if I come across any issues.

louis-lau
0 replies
17h2m

Please elaborate on this, as I'm not exactly sure what you mean. The email deliverability problem is a side effect of false positives in spam filtering. Unless you have a proposal to completely eradicate false positives?

jspaetzel
0 replies
18h18m

for a long time it hasn't been a software problem. It's been a reputation problem, the only recent change is we're more interested in the domain then the IP now.

bo1024
0 replies
16h52m

What do you mean by the email delivery problem? Do you mean setting up DKIM, SPF, and DMARC? It's not that hard, compared to setting up an email server in the first place.

superkuh
10 replies
19h15m

The worst is when they accept the mail but silently tag it spam and put it some place the intended receipient will never see it. Google's gmail is the worst about this. Corporate email isn't email anymore. It's a walled garden / silo like Facebook.

plantain
5 replies
17h17m

I'm certainly not saying it's right - but it works.

Gmail's spam filter (and promotions filter) works with >99% reliability as a user, with really trivial numbers of false positives.

nh2
2 replies
10h12m

Cannot confirm. We get a large number of false positives at our business GMail:

Customer requests for quotes, Paypal/Stripe security messages, lots of other important emails go to spam.

See e.g. https://github.com/nh2/gmail-spamfilters-paypal-security-mes...

For a while now I suspect that GMail has some bug with its own group email addresses:

When somebody sends an email to our GMail group email address team@example.com, it shows up in GMail as "Sombody via team@example.com". Of course, teamexample.com receives both spam and non-spam.

I suspect that when we mark spam that comes "via team@example.com", GMail learns that as "things 'via team@example.com' are often Spam", even though info@example.com is a Google Groups email.

And by now, everything that comes 'via team@example.com' is marked as Spam.

So it seems that when we mark an email that arrived at team@example.com as Spam, Google punishes its own Group email address, instead of the sender.

ipsi
0 replies
2h49m

I think you can set it to be the other way around and have it be "never mark as spam when via team@example.com" - of course, depending on how much spam it gets that might be worse.

hiatus
0 replies
7h3m

For that case you would have to mark the mail as spam in the groups interface, not via the forwarded mail.

ndriscoll
1 replies
16h2m

I haven't found this to be true at all. I've probably marked thousands of linkedin messages as spam, but they still land in my inbox occasionally. I also get random e-commerce related spam from sites I've never heard of. The false negative rate is massive on corporate spam that should be trivial to classify I'd think. e.g. some of them literally have some variation of "this is an advertisement" along with unsubscribe links.

arccy
0 replies
11h33m

I've probably marked thousands of linkedin messages as spam, but they still land in my inbox occasionally.

at this point maybe you should just create a filter rule

jabroni_salad
1 replies
4h17m

Unfortunately a lot of legal protections for unsolicited spam only apply to consumer usage. For b2b, every marketer with the ability to know your email address is entitled to send you as many messages as they want.

If it wasnt walled, it would be completely unusable.

inetknght
0 replies
3h6m

If it wasnt walled, it would be completely unusable.

It sounds like stiffer penalties are necessary for sending spam, both to consumers and to businesses.

rrr_oh_man
0 replies
18h57m

Corporate email isn't email anymore. It's a walled garden

Thank god.

Otherwise I'd drown in cold email spam.

edit: But on a serious note — I'm using Gmail for all companies because I gave up trying to run and administer our own server. It's a travesty that this has become so hard. I feel if you're not on a well-configured Gmail Workspace there's no chance your email gets through, even if legit.

mjl-
0 replies
10h26m

indeed very annoying. my understanding is they don't want to give spammers any signal whether their messages was recognized as ham or spam.

i would love some insights in how smart spammers are in actually leveraging such information. most spam seems to be hammering attempts that don't take failure feedback into account.

in my mail server, messages classified as junk keep getting temporarily rejected with a generic error message. at least the (legitimate misclassified) sender gets a delayed dsn, and finally feedback that a message wasn't received.

it seems many mail servers/services think it's more important to not give a signal to spammers than it is to give a useful signal to legitimate but misclassified users. perhaps they think their classification is really great and doesn't misclassify...

hedgehog
9 replies
14h39m

One thing the April changes break is forwarding between e-mail services. If you currently forward from say an old university address at foo@school.edu to a personal GMail account at bar@gmail.com that will no longer work. This must be relatively uncommon if the major providers are charging ahead with these changes but it's pretty annoying for the people affected.

Ayesh
5 replies
14h21m

Why will it no longer work?

When you forward an email, unless the email forwarder modifies the message content, it should still match the DKIM signature, so it still passes.

hedgehog
4 replies
13h47m

I don't know the details but my rough understanding is after forwarding the next hop delivery will fail SPF.

bo1024
2 replies
13h4m

Yeah, with forwarding, I am seeing DKIM still passes but SPF fails.

tmn007
0 replies
11h46m

We are seeing all the unix forwarders setup a decade+ ago are dead in the water now (have to be replaced with mailing list software)

therein
0 replies
11h11m

This explains why I have been unable to get my 2FA from Adobe.

patja
0 replies
9m

That's my understanding as well, but I also understand that the email will pass DMARC if either SPF or DKIM passes, and DKIM will still pass on forwarded email.

jeffbee
1 replies
4h3m

This isn't a change. SPF has always broken forwarders that don't touch the envelope from address, and that is right and proper. You can still forward mail, but your forwarder must rewrite the return path.

hedgehog
0 replies
21m

It's is a change in mail service behavior, and arguably either a broken policy or broken spec (not sure and don't really care since the effect is the same).

"In April 2024, Google will start rejecting a percentage of non-compliant email traffic and gradually increase the rejection rate. For example, if 75% of a sender’s traffic meets our requirements, Google will start rejecting a percentage of the remaining 25% of traffic that isn’t compliant."

https://dmarcian.com/yahoo-and-google-dmarc-required/

whelp_24
0 replies
9h18m

That seems really significant, email fowarding won't work anymore?

gtech1
9 replies
14h21m

Roughly 50% of my daily Spam comes from @gmail & @hotmail/@outlook accounts.

What exactly are they doing about that ?

yobbo
3 replies
9h54m

Yep. It seems to be from stolen accounts, maybe gathered from leaked account/password lists.

The result is that outgoing hotmail/outlook smtp servers are added to blacklists until they start content filtering their own users.

iamacyborg
2 replies
7h15m

It’s not even stolen accounts. There is a huge amount of cold sales email (spam) sent specifically through Google Workspace accounts. Google do absolutely nothing about it.

sdfhbdf
0 replies
2h16m

Any regular account - a seat - in Google Workspace is limited to sending 500 email per day. I

inetknght
0 replies
3h8m

Why would they do something about it? They're getting paid whether you receive the spam or not.

jjav
2 replies
12h26m

What exactly are they doing about that ?

Nothing at all, because they are too big to care.

Aggressive decentralization is the only way to save the Internet. Host your own email, get everyone you know to host their own email.

samirillian
1 replies
9h32m

Fantasy solution, will never happen.

jjav
0 replies
3h46m

You might be surprised, if you're young enough, to hear that this is exactly how it worked for a very long time.

At my first corporate job, and my second one as well, all email was sent and received directly from/to each persons individual workstation. There was no concept of centralized email server.

(Let alone outsourcing it to some other company, that would have been completely unthinkable.)

jesterson
1 replies
12h50m

They may not - but you fail to properly set up DMARC.

mjl-
0 replies
10h57m

at least gmail.com has a dmarc policy p=none, so a failing dmarc check is not a reason to reject email. i don't think it's that common for small-scale installs to enforce dmarc policies. there are other signals to use though.

plenty of bigfreemail spam is actually sent from their network, they're an interesting target for spammers, and at least some of them put a lot of effort in preventing abuse.

r1ch
8 replies
14h51m

I'm surprised how many big companies fail the one-click unsubscribe test. Whether it's Cloudflare or Akamai blocking the connection, pages that take 5+ seconds to load, pages that require you to sign in or input your email address again... don't be surprised when customers reach for the Report Spam button instead.

nottorp
2 replies
11h57m

I don't unsubscribe from emails I haven't opted in to. So report spam it is.

JoshTriplett
1 replies
10h49m

Same. If I subscribed to it (rare), I'll unsubscribe. If I get unsolicited mail of any kind, I'm not "unsubscribing", I'm nuking from orbit by any means available, including reporting to hosting providers. (This has, on occasion, resulted in the termination of a spammer's account, but the success rate is low.)

nottorp
0 replies
10h40m

I'm mostly talking about those emails with helpful hints and upsells that you get when signing up for a new service. People who make a living from spamming^H^H^Hbulk sending may consider those legitimate.

Ayesh
2 replies
14h18m

I'm using NextDNS with AdBlock list, which is effectively a Pi-hole on the cloud.

The most annoying this is when email senders use click tracking on domains that are blocked by those AdBlock lists. I keep a separate browser instance to copy-paste those links into, but then I have to login again.

I prefer sending unsubscribe emails instead of clicking links. Gmail can automate it.

iamacyborg
0 replies
7h18m

The only reason some clicks still get through is because they’re using CNAME cloaking to mask the tracker.

encom
0 replies
2h49m

Any email I get with click tracking, gets reported as spam.

foreigner
0 replies
10h53m

Agreed, I think of it was a simple UI competition. Customers will do whichever is must convenient: unsubscribe or report spam.

LVB
0 replies
4h46m

I’ll unsubscribe, but now call BS on the “this may take 14 days to take effect…” nonsense in 2024. If I’m getting more emails in a couple of days, they’re getting marked as spam. (Looking at you, TripAdvisor. If you can figure out how to build AI-generated itineraries, you can figure out how to not email them.)

cqqxo4zV46cp
8 replies
16h36m

As usual in any email thread about email deliverability, the amount of FUD in these comments is absolutely mind-boggling to me. I’m not unusually smart or intelligent or capable. I wouldn’t consider myself a deliverability expert. It’s only a small small part of my job. I’ve never worked for any organisation that sells email delivery services to third parties. Why the hell can I understand this stuff, and get it to work, while there are so many people here that very clearly indicate (via what they’re saying in their comments) that they DON’T get it yet have a serious axe to grind?

I’m left feeling like homegrown email delivery is some sort of lightning rod for stuck-in-the-past faux-sysadmin types that can’t get past the fact that it’s not 2003 anymore and lazily / maliciously comply with SPF / DKIM.

IT’S NOT THAT HARD.

tschumacher
1 replies
5h48m

I have the same gripe and in response published the notes to set up my self hosted email on my blog a few months ago [1]. It's really not that hard but yet we constantly see this topic on HN. I understand there are people who've set it up properly and still have their mail end up in spam. Maybe we're just lucky. But there's no need to write long pieces about this going into detail what this tech giant and that tech giant do. All you can do is set up DMARC, DKIM and SPF - that's it.

EDIT: Admittedly this post is also about bulk sending where other metrics like unsubscribe links and spaminess matter. But for the self hosted crew it really just comes down to DMARC, DKIM and SPF.

[1] https://tschumacher.net/self-host-email/

jeffbee
0 replies
3h59m

What I have gleaned from the HN discourse over the years is that the people who are mad about this topic actually are spammers. They want to spam you and they are super pissed off that their little scammy idea isn't working.

nh2
1 replies
9h51m

I agree on the technical part, but:

A problem is that you can do everything technically right, and sitll land in spam, because some big players don't play by the usual rules.

For example, Microsoft apparently has an allowlist for IPv4 -- or equivalenty, blocks all IPv4 by default, until you manually de-list them at sender.office.com. At least I haven't found an IP yet for which I didn't have to do that (self-hosting email for 15 years).

(Imagine every provider did it like MS; you'd be sitting there and filling out web forms with 1000s of providers.)

So you have a technically perfect setup and MS stil rejects you.

--

That said, using some provider to send emails for you doesn't solve deliverability either. There, many customers share the same sender IP. If one of them sends marketing/spam, the entire IP gets bad reputation. In such cases, providers recommend to upgrade to "bring your own IP", which then needs to "gather reputation" [0]. Great, might as well have self-hosted in the first place, as repuation is the only thing I bought the service for.

[0]: Example: https://www.mailgun.com/blog/deliverability/dedicated-shared... -- Especially entertaining is "Use a shared IP if: Your shared IP partners have built a good reputation." As if you had any control over that.

Avamander
0 replies
5h46m

A problem is that you can do everything technically right, and still land in spam, because some big players don't play by the usual rules.

That applies to really all email providers. Part of fighting spam is (somewhat unfortunately) not telling the spammer what you're detecting.

snowwrestler
0 replies
14h56m

Without knowing everyone’s domains and IPs and history, it’s hard to judge competency in a thread like this. SPF, DKIM, and DMARC are important to set up correctly, but doing so is NOT sufficient for good deliverability. In fact it is only a small component of success.

Well-established organizations that have a long history of sending steady volumes of high quality content with low complaints have a huge leg up. So if you work at such a place, or you contract with such a vendor, you’re going to feel like it’s obvious that the DNS entries work well.

gruez
0 replies
10h59m

As usual in any email thread about email deliverability, the amount of FUD in these comments is absolutely mind-boggling to me. [...]

What type of "FUD" are you talking about? The objections in the thread seem pretty well founded (eg. being shadowbanned despite complying with SPD/DKIM, or this requirement breaking email forwarding), and there aren't really any that are against implementing SPF / DKIM.

bongodongobob
0 replies
14h7m

Yeah it takes like 5 minutes. I have no idea what people are crying about. On a tech site no less. This is entry level have the intern do it stuff.

Biganon
0 replies
8h30m

Congratulations for being lucky enough to have big actors accept your e-mails, this is not the case for many of us who do understand and apply SPF, DKIM and DMARC. Guess we'll just try being luckier...?

EGreg
8 replies
17h9m

1. GMail will block your email if you don’t allow one-click unsubscribe. But this is very insecure since anyone can unsubscribe you if you forward your email

Easy Unsubscribe: Implement easy unsubscribe options (One-click Unsubscribe). Gmail users have tools to report spam, unsubscribe from unwanted emails and control their inbox experience. If it is too difficult to unsubscribe from your emails, customers will be more likely to flag your email as spam. Additional links provided in the ‘References’ section at the end of this article.

2. At the same time, Apple’s ITP will start removing all the information from the URL and only leave the domain, if it classifies your site as a “bounce tracker”. This means you won’t even know who to unsubscribe on one click! So all your emails will be blocked.

https://getcake.com/apples-intelligent-tracking-prevention-2...

snowwrestler
2 replies
15h23m

Gmail does not require one-click unsubscribe, what they actually require is that you include the “List-Unsubscribe” header in bulk emails, with a functioning mailto or http target.

If I forward your newsletter, that’s not a bulk email and it won’t include that header.

This is an important distinction that seems to get glossed over in a lot of the coverage and guides about the recent Gmail and Yahoo changes.

saurik
1 replies
8h23m

I guess if you forward it as content but I'm surprised you don't default always to forwarding as an attachment.

snowwrestler
0 replies
5h0m

The default forward action in Gmail sends it as content. Forwarding as an attachment is a well-hidden option in the web UI and I just looked in the Gmail mobile app and could not find a way to do it there.

louis-lau
2 replies
16h59m

Why would apple identify the domain used for unsubscribes as in use solely for being a bounce tracker?

EGreg
1 replies
16h58m

Their heuristics are proprietary dunno

louis-lau
0 replies
16h55m

So you argument here is based on something that hasn't happened, but just something you pulled out of nowhere. Or was this meant humorously?

gruez
0 replies
11h4m

2. At the same time, Apple’s ITP will start removing all the information from the URL and only leave the domain, if it classifies your site as a “bounce tracker”. This means you won’t even know who to unsubscribe on one click!

https://getcake.com/apples-intelligent-tracking-prevention-2...

Your source doesn't actually say that "ITP will start removing all the information from the URL", only that it will "limit it the same way as third-party cookies" and will be "purging website data in such instances".

Avamander
0 replies
5h47m

You can do one-click unsubscribe with headers as well, those aren't usually forwarded by MUAs.

acidburnNSA
7 replies
19h56m

I'm surprised anyone's been getting through at all without perfectly configured SPD, DKIM, and DMARC. I've had a well configured self-hosted personal email server for years and still struggle to get through sometimes, though it does seem to be getting better.

theK
1 replies
19h18m

perfectly configured SPD, DKIM, and DMARC

Just having them perfectly configured doesn't mean that the receiving servers will also see it that way.

Microsoft servers are particularly prone to randomly failing perfectly fine dkim setups for no reason whatsoever.

hsbauauvhabzb
0 replies
15h15m

‘Randomly’ when you just happen to not be part of the mail cartel, they just can’t say that.

shsbdncudx
1 replies
19h1m

Because it’s not just about configuration, it’s also about reputation and a low sending volume you are in danger of getting dropped merely out of not being a well known sender

StayTrue
0 replies
2h58m

“Prove you’re not a spammer by sending a larger volume of emails.” True and diabolical.

xoneill
0 replies
19h46m

I've been wrestling with this for years as well. I hope with these guidelines being published / transparent by the big 3, things behave consistently.

layer8
0 replies
1h47m

DKIM hasn’t been necessary in my experience, and is still not necessary according to the new Google guidelines if you use SPF.

jesterson
0 replies
12h52m

I am surprised people think deliverability consists of configuring SPF (SPD is Seattle police), DKIM and DMARC. Spammers can do exactly same given low entry barrier.

It's important but have very little connection to deliverability in real world.

midnitewarrior
5 replies
18h17m

This looks like a good document, but the author made it political by referencing "Hilary" Clinton and her emails and linking to some Trump stuff. I can't take tech stuff seriously that's dropping in political crap. Go away!

01HNNWZ0MV43FF
1 replies
17h3m

And it's for a company? OP you might want a part-time PR person to review these...

xoneill
0 replies
16h57m

Removed! Yeah, bad attempt at humor. And certainly not done in support of said candidate.

xoneill
0 replies
16h59m

Very valid point. I just removed my poor attempt at political humor. It was not meant in support of any candidates.

jesterson
0 replies
12h46m

Are you a truth seeker or attention seeker manifesting your political preferences instead of focusing on what article actually discusses?

that's a rhetorical question - you already answered it.

g4zj
0 replies
18h1m

Yes, this put me off enough that I was no longer interested in reading the article. The silly reference is one thing, but was a link to that YouTube video really necessary?

kureikain
5 replies
19h40m

I run an email forwarding service[0] and it's damn hard to get into inbox of any major provider if SPF/DKIM aren't config properly. DMARC or ARC might be optionally but an email without SPF/DKIM, good luck having it hit any inbox.

Office365 is the toughest, email just randomly land on spam no matter what I do. Icloud, actually it's ProofPoint is tough sh*t too.

So I'm so surprise these guide just pop-up now like it's a new thing.

---

[0]: https://mailwip.com

jmb99
1 replies
15h45m

I’ve been running a personal mail server since 2016, and I’ve been struggling mostly with Microsoft off-and-on. I haven’t make any changes since setting up DMARC, SPK, and DKIM in 2016, but I’ll still sometimes randomly get blacklisted for 2~30 weeks for seemingly no reason, and then get unblocked again for seemingly no reason.

It’s recently started happening with iCloud too. For 5.5 weeks any email I sent would either get bounced or land straight in Junk, until yesterday when the powers that be decided my mail was worth delivering again.

I’ve somehow never had an issue with Gmail or Yahoo, and of course never with any other non-big-three mail servers.

dugite-code
0 replies
12h19m

Could be your IP block gets the odd spammer or you don't send enough emails and their servers resets you IP address reputation, meaning you essentially become a new email server.

xoneill
0 replies
19h23m

This was my initial thought too. These guidelines have been around for ages, and just now being officially implemented in 2024.

jesterson
0 replies
12h49m

ProofPoint is indeed disgusting service. Wonder how they proliferated into corporate world so much and so fast.

77pt77
0 replies
19h10m

but an email without SPF/DKIM, good luck having it hit any inbox

And that's a decent thing.

Someone that doesn't do that has no business sending email.

The problem is that even doing that and even sending only a couple of emails a day is still usually considered SPAM.

xoneill
3 replies
20h34m

Just a freindly reminder. I wrote a thorough guide on email hygeine here, includes validation tools to help troubleshoot & straighten things out.

ryantgtg
2 replies
19h39m

At an even more basic level than what you described here, I recently improved my delivery rate by adding a name to the transactional emails that we send (from: 'My Name <admin@foo.com>') where previously we didn’t have a name. As well as cleaning up subject lines - where previously we included some abbreviations. Providers like mac.com were previously soft bouncing some emails, and now they seem to accepting them.

Hotmail/yahoo/aol all still seem to shove our transactional emails into spam pretty often (judging purely on the amount of those users who fail to confirm their accounts).

xoneill
0 replies
19h17m

Thanks for the info. For years, I've been using trial & error / reverse engineering approaches to improve deliverability. Frustrating!

patja
0 replies
18h43m

mac.com is the worst. I ban that domain from new registration now.

And Apple sends no dmarc reports nor do they implement any actionable feedback loop.

cj
3 replies
18h17m

The thing that kills me about DMARC is how often is fails with Microsoft specifically. And also with any use case involving the recipient forwarding mail (which breaks SPF alignment)

I want to follow best practices it recently changed p=quarantine to p=none after fear that legitimate emails aren’t passing DMARC despite properly configured DKIM and SPF.

Hell, I would love p=reject but not until recipients fix their incoming mail servers to handle edge cases like email forwarding breaking DMARC

louis-lau
2 replies
17h9m

Senders that apply dmarc and want their emails to be forwarded should use dkim. Forwarding a dkim signed message doesn't break dmarc at all.

zinekeller
0 replies
13h48m

how often is fails with Microsoft specifically

This is the most important part. Exchange (due to its history as an X.400 server, not as an SMTP server) does sometimes mangle the message to the point that DKIM simply breaks. This both breaks origin-incoming and forwarded messages.

BTW, Apple also sometimes mangle messages that it fails DKIM, although I do not know why is this the case (as I doubt they use Microsoft Exchange for their mail service).

mjl-
0 replies
10h15m

this is a long standing problem with mailing lists. they are often configured to add a "[...]" prefix to the subject or add a footer, breaking the dkim signature. this leads some more recently updated mailing lists to always rewrite to their own "message from" header, so they control dmarc alignment for their messages.

for incoming email on mailing lists i'm subscribed to, i don't enforce the dmarc policy. i think this is what the parent post hints at. i'm not sure how easy this is to configure with the various mail server software out there. i'm also not aware how you would configure this with sieve scripts (i looked, didn't find it, but it seems like a basic case).

if you're running a mailing list, hoping for all subscribers to not enforce dmarc policy enforcement doesn't seem like a great strategy.

the forwarding case should be easier to keep working.

briandear
3 replies
10h40m

I wish we could solve the unsolicited SMS problem.

jeffbee
0 replies
4h1m

I haven't got any since I switched to Android. Same with spam calls. Their crowdsourcing really works.

elric
0 replies
6h8m

I have received exactly 2 such messages in the 25+ years I've had a mobile phone. I imagine this is a regional thing, and that where you are eithet mobile networks are insecure, or operators are in cahoots with spammers?

ai_what
0 replies
4h20m

I totally agree, and also unsolicited calls. Right now the only options that I'm aware of are:

1) Blocking numbers (which is pointless because they rotate them and spoof residential numbers).

2) Whitelisting numbers and blocking everyone else (this would cause me to miss legitimate calls).

3) Blocking entire ranges (this doesn't work because of the spoofing).

4) Using one of those spam screening services (currently looking into this). Still a bit concerned about missing valid calls and the privacy issue with this.

Avamander
1 replies
5h51m

Use rspamd?

mmd45
0 replies
5h0m

thanks, that seems promising. i will look into it.

solatic
1 replies
10h52m

No mention of BIMI (either for or against)? I'm surprised...

Avamander
0 replies
5h49m

It doesn't affect deliverability in a measurable way compared to DKIM/DMARC.

nanidin
1 replies
13h8m

I ran my own mail server for more than a decade. Same IP the entire time, never sent spam (for personal use only.) Finally threw in the hat last year and moved to a paid service - it was a pain to tell every person I sent mail to to check their spam box and mark me as not spam or add me to contacts. Beyond that, gmail smtp servers kept getting onto spam blocklists, so I wasn't receiving mail from gmail at times.

dugite-code
0 replies
12h29m

Gmail and Microsoft require a certain volume of email hitting their servers otherwise they forget you exist. I ended up switching to using Amazon's SES service to keep my selfhosted server running.

I use seperate emails for each service much like a seperate password so I'm heavily invested in keeping my own server at least for recieving.

inetknght
1 replies
3h24m

Spam is indistinguishable from malicious content.

You did not sign up for the "newsletter". Your email address was harvested and given to malicious actors hell-bent on screwing you. Clicking on anything will take you to a website where your best interest is not at all what the company is going to do with your information. At best you might just remove one source of junk in your inbox. At worst, you end up clicking on something that turns out to install malware on your machine.

So what should you do?

1. Don't click on unsubscribe links.

2. Click the spam report button

3. Stop using big email services that ignore spam reports. Gmail panders to other big businesses by letting them spam you without giving you the option to blacklist the entire domain yourself. Malicious content will continue to enter your inbox until you move to an email provider that takes your privacy and security seriously.

DeanGadberry
0 replies
1h42m

Fair assessment. Which email service provider do you use?

deadbunny
1 replies
19h39m

So what any competent sysop has been doing for years?

louis-lau
0 replies
19h30m

Sometimes it's not about competence but about priority. If email is something a business does as a side thing and not a core thing, and if email keeps working without changing anything, there's no need or priority on setting up newer things. From a priority perspective at least.

But yes if it's a recent setup, or email is a core part of the product, any competent sysadmin should have been doing this.

willyt
0 replies
3h28m

Speaking as someone in an industry that receives a lot of unwanted and seemingly un-unsubscribable marketing emails. I have never ever bought anything from a company that has sent me an email cold. I have my inbox set to show the first two lines and I delete them without opening them pretty much all the time. The only thing marketing emails do is annoy me.

paulnpace
0 replies
6h52m

I wonder how this ends up impacting government agencies and especially courts and law firms. My experience has been all three struggle with these things.

keybored
0 replies
7h20m

I’ve gotten some emails from Gmail about delaying my emails to Gmail users because I apparently send too many emails. I use git-send-email(1) which might send a cover letter plus X patches right after each other. These Gmail users are then in the CC. So I’m not a mailing list. The email list is the To recipient.

I’ve been wondering if this was the cause. I don’t send out 5000 emails (I’m not 10X). But there’s this part:

While these guidelines primarily affect bulk senders, senders with less volume per day can also be affected if they are not adhering to these guidelines.

I haven’t looked into it yet but I guess I should.

I use my own domain and I’m hosted by a not-Gmail provider.

andimm
0 replies
6h21m

I think you mixed envelope (RFC5321) and headers (RFC5322) in your text.

The domain name in the From: field in the email envelop header is inspected and aligned with other domains authenticated by either SPF or DKIM:

The envelope does not have any header, the headers are in the content/body of the email. Also your screenshot of the "Here’s an example email envelop from an organization that passes all of the email security guidelines:" are the mail headers and not the envelope information.

Great presentation on this topic from dmarc.org

https://dmarc.org/presentations/Email-Authentication-Basics-...

Animats
0 replies
13h32m

Then there's the other side - receivability. IDrive is supposed to send me an email each day reporting backup status as seen by the backup servers. Those messages have been flaky since mid-February. Logs indicate the backups run; it's just the completion emails that fail.

Their support people blame me, although they admit others have the same problem. They're not using a mail delivery service - the emails come directly from an IDrive server.

They're sending to my web site, which forwards to my personal address. There's no filtering at the first stage, and a division into Accept/Greymail/Junk at the next stage. Neither Google nor Yahoo is involved at any point.