I manage IT at a mid-size business. At least once a month, I get asked to release some incoming email from quarantine that got sent there because the sender's SPF record is wrong or outdated and doesn't include all the email services they actually use. (What this really tells me is how many small businesses are out there running with no in-house IT expertise or support of any kind.)
I don't do whitelisting. Instead, I always reach out and offer to help the other party correct their SPF record.
It happens often enough that I wrote a script in Racket that will generate the email for me and paste it into the clipboard [1]. The email tells them exactly what they need to change, and links to docs from their current email provider (so they don't have to trust me about edits to their DNS).
[1]: https://gist.github.com/otherjoel/6b8bf02f6db6e0c47ba6bca72e...
And at the same time, I regularly get Spam/Phishing with perfect SPF, DKIM, DMARC, etc. The domains and IPs they use might get blocked within a day, but of course, these people have no problem getting others.
And although I have set up my MTA perfectly, my mail gets refused by MS/t-online/etc., because I don't have enough "sender reputation". In e-mail, we have an oligopoly of a few big mail providers, and in the end, they decide which mail gets delivered and which isn't, and to me it looks like they give a rat's ass about SPF and DKIM, and probably rightfully so, because most spammers are probably better at configuring MTAs than your average mail admin.
t-online uses a global whitelist, which is pretty stupid for e-mail. sometimes it helps contacting them, other times they refuse to resolve it for arbitrary reasons (not because of actual spamming)
t-online told me I needed an imprint on the website that's reachable under my domain. Seems to be some misunderstanding of German law (German commercial websites need an imprint, legally, but t-online also apply this requirement to private domains).
It is more complicated than that. There are more criteria for when you need an imprint:
(1) any kind of journalistic content on your site
(2) any kind of financial gain from showing ads or making ads
(3) organizing any kind of group of people active on German territory
(4) running a business website
There might be more, but those are the ones I remember from reading the paragraphs a while ago.
And these are, of course, vague, which means that even something like "my favorite restaurants in Berlin" could be considered an ad, or any kind of comment on politics might be considered a form of journalism.
I dislike these rules, because they basically kill German blogging scene. Not so many people want to run a blog and have every idiot on the Internet know their personal address. And few bloggers want to rent a digital office or actual office, that will send mail to them (an indirection). The German law in this respect is terrible and working against a free Internet and against freedom of voicing your opinion. It works greatly in favor for tech giants, because people resort to putting their blogging on Facebook, Instagram and other disservices. It is very anti-decentralization.
Last month they unblocked me even though the website is blank.
T-Online has a simple whitelist approach, and it is usually enough to just drop them an email. I did that back in 2014, asking for my private mail server to be added to that whitelist, and I received a positive answer within a few hours.
Maybe that was possible 10 years go, they now require that you put up a web page for your domain with a valid German imprint (most importantly: your full contact information).
To be fair, I had that in place 10 years ago (and if you already have a mailserver, it's trivial (as in: MUCH easier than to set up a mail server) to host a small imprinted HTML page).
On some level I can see it being a benefit to the big providers to only accept email from other big providers, as it would incentivise people to buy email services from them, because only email sent from the big providers would "work".
What would you say the normal reception you receive from this email template is?
I like the idea, but I would think sending a technical email (with industry-specific acronyms that you don't spell out!) to a business that has no in-house IT would just be ignored in most cases.
Well if you read the template, you'll see I start out with a non-technical explanation, advise that they forward the email to an IT type person, and offer to help in any way I can. Then I put in a "More info" heading further down with all the details and instructions.
Overall I'm pleased with how well this approach works. When people realize that their email is getting stuck in spam filters because of a problem on their end, they're usually motivated to get it fixed. Sometimes it gets sent to an owner who had barely enough tech mojo to stand up a gmail account at a custom domain, and even then the instructions are usually simple enough for them to follow.
I read the template, that's how I spotted the acronyms that weren't spelled out. Like DNS on the second line, before you recommend forwarding.
Interesting, I definitely would have thought it'd be ignored more often than not, but I might have to look into rolling out something similar. Thanks for the idea.
I think you're being a bit unfair here.
If you don't even know what DNS records are, I'd imagine you'd assume you don't have access to them and so forward them to the IT person as suggested. But sure, maybe he could also add ", or don't know what they are" to this line.
I'm not trying to be unfair or critical or anything. My first question was genuine. I intended my note about acronyms to be just that: a side note. The response I got was "If you read the template [...]", which it should have been pretty obvious I did. Then I got an explanation of the template as if I hadn't read it, which was a bit patronizing.
I think it's a good idea (and said so twice!). I was curious what the reception was like.
Sorry if my comment about acronyms was too much. It is a pet peeve of mine to see acronyms not spelled out, especially technical ones in a document intended for non-technical people. I didn't intend it to derail the conversation. Obviously it was taken in a way more critical way than I had intended -- my fault.
Sorry I didn’t pick up that you had read the template. I was just trying to give context for my answer without assuming or requiring anyone who might read it to have scrolled through all the code.
I see no problem here.
Cool stuff! BTW, what's up with the license?
Including the whole license in the gist was probably overkill, but I particularly like Blue Oak for permissive licensing and would like to see it more widely used.
https://writing.kemitchell.com/2019/03/09/Deprecation-Notice...
I'm not sure I like it. Like, what if the notification of notice was incorrect? You lose your license anyway?
By default under most open source licenses, particularly permissive ones, violating the terms even by accident ends your license instantly, with no notice whatsoever.
Including a fair, common sense path to forgiveness severely limits legal risk for users, and is one of the things I like about the Blue Oak license.
Is it osi approved?
Hey, English is not my first language so I'm surely missing something here but:
This sounds like the license specifically allows you to infringe on the contributor's copyright.
It's a permissive license. The licensor is saying “I own copyright in this work, which gives me the right to give you permission to do whatever you want with it.”
Does the script handle macros in SPF?
I've had a couple of other-company-IT-admins tell me that my MX is jacked because I use hosted SPF via proofpoint, and when they look up my SPF it looks like this:
"v=spf1 include:%{ir}.%{v}.%{d}.spf.has.pphosted.com ~all"
A surprising number of mail admins don't understand SPF macros.
If I’m reaching for the script, it's because I’m already in a scenario where Proofpoint has quarantined legitimate email for failing SPF checks (we use Proofpoint too). So the script itself doesn’t do any analysis of the existing SPF record. It just shows them the existing record and tells them how to fix it based on the sender's IP for the email in question.
In defense of those who haven’t read the RFCs personally — I can count on one hand the number of times I’ve seen SPF macros in the wild, which holds true if I included yours.
Interestingly all Proofpoint customers too.
I’ve seen it more common to isolate services to subdomains and specify subdomain SPF records rather than use macros. This is my preferred approach.
I’m not hating on the macros. They’re just seemingly very rarely used. I know they’re on the table but I haven’t found a compelling use case in my own deployments.
How do I make use of the gist?
I do
But afterwards none of the definitions are available in the Racket REPL. Maybe I need to (require ...) something? I do not see a module definition, that I would need to import and fails.If you’ve just run "racket" from the command line to get a REPL, you would use
to import all the bindings from this file (assuming it's in the same folder you ran racket from). The “Module Basics” chapter of the Racket is a good quick explainer of how this stuff works in Racket: https://docs.racket-lang.org/guide/module-basics.htmlAnother method would be to open the file in DrRacket (or VSCode or Emacs or whatever editor you have set up with a Racket plugin/lang server) and just "run" it in the REPL.
There are comments at the top that explain how to use it once it's loaded:
Personally, I have this file incorporated into a larger package (not published anywhere) for producing canned responses. With that package installed I can do this at a command prompt:Awesome, mind if I send you an email solely to test if it gets through or if I get to be the recipient of your awesome script? ;-)
Sure, I think?
I work at an IT provider - we see this daily. Have to whitelist to keep the customer happy, usually the other end is a 1 or 2 person business with an old hosted Yahoo system or similar
or misguided?
(I know the pain.)
I was on the receiving end of an automated version of this. However, when I looked into it, it seemed the problem (SPF record required more than 10 DNS lookups) was fairly common. I don't seem to have any other deliverability problems and my email and DNS is managed by some big hosting company so I assume it's not a real problem and didn't fix it.
It’s a real problem. The solution is just to isolate each service on its own subdomain. The only thing that should be listed in your top level SPF record is the corporate email for the domain.
That is really awesome. It can be easy to miss setting up SPF on every new tool.
Yeah, the only thing you need for simple SMTP services is:
- with DNS, the simplest as possible SPF record.
- Without DNS, (aka pure IPv[46] SMTP), your have implicit SPF: instead of querying the DNS and parsing the SPF record, you parse the mail header to check "reply-to/from/etc" fields (the appropriate fields) for the sending SMTP IPv[46] address, that to perform spam scoring.
If we care about keeping open protocols like HTTP and SMTP alive, we need to overhaul DNS.
Or at least create a simplified common abstraction layer.
It’s the most inherently user-hostile thing I’ve ever encountered - and I’m only just now starting to understand it, even though I’m almost 20 years into dealing with it.
Neat! I must try this.
My favourite are queries "why are you rejecting my email?"
Ehh... Because your dmarc policy told us to?