I run one of the largest email sending services on the internet. I have been living the “mess” of internet email for over 20 years.
Here’s the thing: despite the internet’s email system being complex and confusing and riddled with problems, it is universally adopted and interoperable. No other open communication tool can boast of email’s massive interoperability.
Any new system will face the impossible burden of winning hearts and minds while the old system continues to chug along with billions of annual R&D spending and dozens of conferences full of smart people working on solving problems.
Even if “MX2” can peacefully coexist in the DNS, why would anyone spend the millions of dollars in engineering effort to move while their teams are busy building the latest layer that’s been invented to patch over the current system?
By all means if you want to propose a new email system, show up at IETF or M3AAWG and make a bold proposal. Someone will buy you a beer while they explain why you are much better off getting into the mud pit with the rest of us and working on the next pragmatic fix to keep things rolling along.
I run several of the smallest email sending services on the internet. Been doing it for 26 years or so.
{rest of your comment ... verbatim!}
Email doesn't need fixing or a v2. I've run GroupWise, Exchange, Lotus (various flavours but never peppermint) and others too. For me, for little systems: Dovecot (IMAP) + Exim (MTA) is a golden combo.
Email_v1 itself is just complicated enough without being too funky. You bolt on TLS/SSL and the rest. MX records are a simplified form of SRV records (I think MX came first but the point holds) and remove the requirement for gateway load balancers and clustering technologies if you don't want to do all that stuff.
Nothing is perfect but email does deliver a lot more significant messages than any other method from before or since its inception.
If email didn't need fixing, spam wouldn't exist. There's more spam than there is legitimate email traversing the internet. That is a problem worth solving.
You could solve it with existing infrastructure to some extent, eg. your email address is actually a cryptographically generated guid rather than something easily guessed or harvested. If you combine that with a background handshake procedure for introductions, so that all of your contacts get their own guid alias mapped to your canonical one, then you can revoke any of those if they get compromised at any time. Spam is effectively solved.
This is basically like the web of trust, but for email.
Spam is not a technical problem, it's a societal problem. As usual the tech reflex is to find a technical solution but it is mistaken, once again. Societal proplems require societal solutions, not more tech. The only thing you will achieve with more tech is more segregation between those who have and those who don't, you'll create more issues than already exist
Spam is 100% a technical problem. Any "societal solution" will be orders of magnitude more expensive and less robust than technical solutions.
Junk mail is a lot older than the Internet.
Email being so cheap and easy is is a significant component of spam, which is a technical problem to a degree. Spam on Signal, text message, voicemail, Discord, etc… is significantly less present for various reasons (cost, complexity, etc…)
:sigh: speaking of getting into the mud pit...
The other side of that balance is that capitalism creating artificial incentives for bad behavior is a significant component of spam.
Here, you're kicking the problem further down the road, though, to another known attack vector: Directory Harvest Attack[1].
In this case, though, the directory (presumably) contains the guid mapping (which - by definition - would have to be a different guid than the object) and would have to process parsing these guids against the users. (This already occurs on recipient receive for some SMTP servers [just before BDATA/DATA] via the email address).
What would one bad email to an email guid do? Would it force rotation of the guid[s] throughout the entire forest? If so, how would that be communicated externally? How would you communicate it for just the one address, if you just changed the one guid?
Would you, instead, have to keep a guid history to check against -- or lose all of the email between possible compromise and the sender's database update? Would you just keep it in the Transport Queue, until manual intervention could check out email between the possible compromise of the guid and new mail would be received for the new guid? That wouldn't scale for large enterprises.
Keep in mind that nothing has to be sent for recipient validation to occur. The SMTP Server[s] just respond[s] to the recipient block with the next step -- but the caller doesn't have to complete the SMTP negotiation from this point, they already have validation if the addresses (even these proposed guids) are valid.
Tarpitting is somewhat of a viable option, here, but it isn't foolproof.
[1] - https://en.wikipedia.org/wiki/Directory_Harvest_Attack
Dictionary and brute force attacks don't work against cryptographic ids, so I don't see how this is relevant.
I assume you mean, what would happen if you received a spam message and had to revoke a guid? First, revocation means the guid is no longer valid and to any incoming message, so it acts as if the guid simply doesn't exist.
Second, the idea here is that every entity gets their own guid designating you, so the same guid is not known by more than one entity. This is the purpose of the handshake protocol during introductions. If A and B know each other, B and C know each other, and A and C want an introduction, B triggers the introduction protocol which mints new guids for both A and C that are then exchanged with each other. This can happen transparently without the user seeing what's going on under the hood. Revocation is just a mark as spam button, and introduction is triggered by CC'ing more than one person in your address book (introduction is the trickiest part).
So if A gets a spam message from C, you just revoke the guid sent to C and you're done, any message from C now acts as if A's address no longer exists. This doesn't affect any connections to anyone else.
If B's guid for A is compromised in some way, you can trigger the introduction protocol again to mint a new guid after the compromise is resolved, then revoke the old one.
There is simply no way for spam to gain a real foothold here: they can't guess ids, and if they somehow obtain someone's address book, those addresses are valid only for one or two messages at best, before it gets revoked. The revocation and introduction protocols can happen using the existing protocols in a few different ways, like by exchanging some message types that are not seen by the user. There are definitely some details still to work out but I don't see any real roadblocks.
The only real "problem" is that now all email addresses are effectively private, eg. no globally addressable emails, which is not great for business purposes like info@mycompany.com. You could of course keep running the old email system for this.
The first paragraph sounds exactly like paper mail.
And yet, we didn't solve it. It just got worse.
That seems region specific. I don't get paper spam in Australia. I did get some for a week, until I put a "no junk mail" sticker on my box, which is respected here. It's less of an issue of paper mail and more of applicable regulation.
If your email address was a secret known only to your existing contacts then how does someone reach you if they don’t already know you?
No. Big email server all over the place might drop your emails for whatever reason is and they won't tell you.
Question is whether a new or updated protocol could force users such as "email providers" to change their behaviour.
Here is something to downvote, a series of questions:
What if email were "pickup only" _from the sender_. Not from some intermediary recipient, e.g., an "email provider". What if the the recipient had to identify acceptable senders before they could "send" mail to the recipient. Arguably this already happens every time an email recpient gives out their email address to some email sender. What if the sender did not "send" mail but instead uploaded it to host run by the sender and accessible by the recipient. Then the recipient retrieves the mail from the sender.
In the past, one large email provider had an RSS feed for a user's inbox. What if the RSS feed is not provided by some third party "email provider" but by _senders_. What if the feed indicates whether there is new mail waiting to be retrieved by the recipient. Arguably, something like this already happens, albeit using third party intermediaries, for example as millions of people use non-public webpages on third party websites to communicate with each other, instead of using email. Recipients check these pages for comments or messages from "senders".
A simpler idea that requires no changes to any email protocol, which I have tested successfully on home network, is for sender and recipient to be on a peer-to-peer overlay and run their own SMTP servers, like the original internet. The sender SMTP server communicates directly with the receiver's SMTP server, not a third party SMTP server run by an "email provider". Obviously, sender and receiver should not invite anyone onto this network who they do not know and from which they do not want to receive email.
The fundamental problem with email is that personal, non-commercial email is mixed with commercial email, mail that is selling something. That's beneficial to marketers, but not email users. Any change to email that threatens to exclude the unsolicited, commercial email will be opposed fervently.
The pickup idea sounds similar to how newsgroups work.
Newsgroups are full of spam.
I doubt there will be a technology that can reliably exclude unsolicited or commercial email. How will the system know, what's unsolicited or unwanted? It can make a guess and that's what the big ones do. But it won't get better than this. I don't think there will ever be an alternative where this could be opposed.
As for unsolicited, this is already taken on by the GDPR and if you're a company that wants to sell their stuff in the EU, you pretty much have no choice than to adhere to these laws.
Anything from an address not in my address book?
Congrats, you've invented DJB's Internet Mail 2000[1]. Definitely a good proposal for moving the burden of spam back to the spammers but I don't think anyone took the time to seriously consider it.
[1] https://cr.yp.to/im2000.html
ideas worth stealing!
Over 15 years ago, I took the time to seriously consider IM2000 and I am still asking today "What if..."
Protocols aren’t going to change that. Email has to be open or it won’t work. But being open means there is abuse. And abuse means there is reputation. And reputation… means you will be blocked sometimes.
but you chose that service, so what's the issue?
Also, they would school them on actual-world problems in the process:
- You can't wait until you receive the entire body to be able to compute a signature to then validate the sender as your first line of defense. It is just too expensive and opens you to DDOS attacks. People use IP reputation as a first line of defense because it is cheap, not because it is good.
- You cannot enforce people's behavior through RFCs. I can assure you that random guy next desk will not care about your "this is a top-posting-thread" header and bottom post there. Even if she has to manually copy/paste things around.
- Likewise, auto-generated plain-text versions of HTML (or other rich-text formats) are no better than what screen readers can achieve. Most poeple won't bother writing that alternate version, meaning the obligatory alt text is now less useful than when it was optional and only people who cared inculded it.
- Your largest client may not update their e-mail infrastructure to comply with the latest standards. If that happens, you don't tell them to update or otherwise you won't be answering them because their e-mails go to spam. You do whatever is necessary to ensure that their e-mails don't go to spam. Business always comes first.
We should move away from having a single mutable body for email. It should be a series of immutable messages that reference the message that it is replying to. Each message can contain a hash signed by the private key for the domain that wrote it. Then when you write your message it just gets appended to this chain.
How it is shown is up to the email client so that it can be done in the best way for the user.
What about responding inline to parts of a message?
We do it like Hacker News. It's just another message, with > indicators. Globally, inline replies are A. Rare and B. Often used with prank intent (i.e. you can make it look like you're replying to something they didn't say).
Not sure about globally, but on most mailing lists I read they are used quite often.
Me being able to prove that I wrote something is good. Other people being able to prove that I wrote something… it's good under many circumstances, but not in general.
Hypothetically, while I’m no expert:
1. Could a future protocol require an immediate initial message (a “hello”) stating exactly how much content will be sent, and until the “hello” is sent, it’s limited to, say, 128KB before the connection is immediately terminated? (And of course, if the content exceeds the declaration, termination and immediate IP temporary ban, safe to do as this is an obvious violation of a new spec?)
2. The goal is to make it easier for the email client which by itself will encourage good behavior. There’s also no requirement for the messages to all be in one massive blob.
3. The goal is that it would be automatically created by the client. For personal emails, this is easy. For enhanced HTML emails, that is where the requirement comes in. Email providers can come up with their own ways of enforcement from there (I.e. “if it’s only one sentence, you obviously didn’t do it”), though I get your point and that would become messy unofficial spec again.
4. Could a future emails system have versioning, allowing the server to clearly communicate (“Hello, I implement MX2 v3.1.”)? In addition, a business can obviously make their own settings that original email alerts do not go to Junk in their business mailboxes - but they do know they’d better get on it or their messages to clients might go to Junk.
SMTP already has the BDAT command where the size is sent first, and arbitrary bytes can be sent (unlike DATA).
SMTP already has versioning through extensions.
If you're banning an IP for exceeding a processing resource limit please keep the ban short. Presumably you can afford to process the first 128KB of one bad message per six hours, for instance. There should be no need to make a month-long or permanent ban, and these just hurt interoperability if the sender realizes their problem and fixes it, or if the address is reallocated.
Trying to limit data between the hello and the email data is futile, since the attacker can just flood you with random packets no matter whether you told them to stop (closed the connection) or not. You can only limit things you have control over, mostly your own memory usage, and how much data is accepted into more expensive processing stages.
Why would it take millions of dollars to move?
The Gmail engineering team is perhaps 500 strong based on various estimates that try to exclude team members who work on Google Workspace more broadly. At Google’s median pay of $300K, the cost is very approximately $150M/yr to engineer the stuff that makes Gmail work. That’s before accounting for infrastructure specifically required for engineering efforts, such as training models.
Let’s speculate that migrating to a whole new email standard would eat up 10% of the team for a couple of years, giving time to build and adapt a software stack that can run at Google scale on the new standard. That’s a $30M spend.
While this figure is peanuts for Google, consider that during the span of time that engineers would be transitioning to “email 2.0,” the full Gmail team would continue to work on just maintaining “email 1.0” and adapting all the new standards that continue to make the existing system improve.
Why would anyone at Google want this disruption unless email 1.0 was so broken that it was hopeless?
Email is amazing and underappreciated. The federated social network that everyone keeps trying to invent has been there all along. If more people controlled their own email address, it would be nearly perfect.
I’m not in that space but it sounds familiar. In my experience there are a few phases in fixing a problem:
- Noticing the problem
- Finding something better
- Finding a way to migrate between the problem and the fix
The hardest is the last bullet. It is easy to say “things would be better if X” it’s much harder to build a roadmap and buy-in on how to get to X.
I think it all depends on the right pitch:
As a replacement for current email only, you're probably right in that nobody would care enough to do the significant investments necessary in a reasonable timeframe, similarly to IPv6 (and the pain there is arguably even greater than for email).
As an alternative to something with the semantics of what companies currently use SMS for (i.e. having ostensibly higher reliability and security), I think you'd see a lot of interest.
https://xkcd.com/2347/