I received the following idea via e-mail from Douglas Arndt following my MPR appearance the other day, talking spam.
“Perhaps I’m missing something, but SPAM seems to be a very easy to solve problem. It would be solved in two simple steps.
1) Individual SMTP server owners would change thier server to “authenticate” the sending and reply-to address when the message is received on port 25. This is to say that the server turns around and ensures that the sending and reply to address user1@mailserverb.com is a valid user on mailserverb.com. This is simple and it could easily be implemented in today’s protocol. If the user is a real user, simply accept and deliver the message.
ISPs, like Visi, AOL, Hotmail, e.t.c. could get users onto this service by simply offering it as “registered e-mail”. I would think that users would flock to the registered service in droves. Most “normal” e-mail would simply pass through this service. This e-mail to you would pass through because doug is a valid user on the server reliablesites.com.
This would immediately close down open relays, intentional and unintentional.
2)To the extent I get spam from a valid user on a mail server AND I send a message to unsubscribe AND the user sends me another unsolicited message, I can send the evidence to the real-time black hole list and the server or netblock can be added.
This approach adds the right incentives in all the right places. A legitimate bulk e-mail sender is incented to remove you from their list, have valid send-to and reply-to addresses. The server provider is very incented to not have their IP or netblock added to the real time black hole list. Hosting providers are very incented to police their own community as
it would be a disaster to have your hosting center’s netblock added to the black hole list. Finally, users are incented to move to registered e-mail because they know it’s a real user that’s sending the message.
The government will simply make a disaster out of this if it gets more involved. The private sector can solve it if they want to. Perhaps the only role the government should have is over the black hole dispute/arbitration process. I’m unsure if this is even needed though. If Visi’s entire netblock is added improperly, and as a result Visi is shut down for a period of time, Visi would have a valid legal claim against the black hole administrators.
Just a thought.
Kind Regards,
Doug Arndt”
Think he’s right? Will the government — say, in Lessig/Lofrgren‘s proposal — really just mess it up? (Could it be any more messed than it is at present)?
John Palfrey 
Any time someone says that solving the spam problem is easy, this should set of blinking red lights in your head. Economics 101: if it was so easy, someone else would have done it by now.
Problems with these suggestions:
* If the relaying server in #1 is behind a firewall, the accepting server cannot verify anything. (Please note that the relay in this example does not have to be serving as an MX server for a particular zone and as such it would be perfectly normal to have it behind a firewall; in fact, in large organizations, this is the norm).
* Additionally, number 1 was invented many years ago: it’s called “ident” and is basically obsolete at this point because people realised that it wasn’t reliable and posed all sorts of security problems. I won’t go into that now.
* Number 1 requires that all mail is relayed by the MX server associated with the sender (or, at least, that the relaying server has knowledge of the valid addreses accepted by the sender’s MX server). One of these is impractical and the other breaks the idea of loose coupling in SMTP. Neither has much benefit.
* While we’re breaking SMTP, let’s talk about the inability to directly connect to a zone’s MX server. Now we require all mail to be relayed.
* Number 2 almost advocates blacklisting. I promise you that blacklists are a bad idea. Please do not advocate them unless you have a very deep knowledge of them. Read my paper for more info:
http://theory.whirlycott.com/~phil/antispam/rbl-bad/rbl-bad.html
This is a complex issue and there are really only two choices:
1 Fix what we have
2 Throw it all away and start again
Throwing it all away is a huge task that will probably take over 10 years. I just can’t see that happening, practically speaking.
Keeping what we have requires a systemic approach to this complex problem (see “systems thinking”).
Here’s a statement… who agrees with this:
SPAM is the largest problem facing the Internet today.
I think I might agree with your last statement, or at least be willing to say that you’re very close — and I feel like it’s getting dramatically worse at the moment.
Also, to be clear: I’m far from advocating blacklisting. I think the RBL and its ilk are at once intriguing and incredibly frightening, in their way.
Lessig’s “tag it and put a bounty on it” solution — also limited, for obvious reasons — is the best one out there at present.
-JP
I actually think Larry Lessig’s bounty idea is a good one, but I do not believe that there is any one single silver-bullet answer to this problem.
It’s a complex problem that involves (1) economics, (2) technology, (3) law and (4) social behaviour and what I’ve been thinking about lately involves a solution encompassing all of those elements.
The proposed solutions that I have seen so far have come from people with backgrounds in one of those areas and their proposals have fallen short in the other areas I mentioned.
Also, I re-read your post and you were indeed not advocating blacklisting. Thank you for clarifying that for me.
It’s the biggest problem out there on the net today (ok, so at least in the top 3!) and some people are still saying, “Just hit the delete key! What’s the big deal?”
Hi John et al –
I agree that the original suggestion, as written, is impractical – but it got me thinking. Why couldn’t ISPs provide a “registered mail” service? It could involve simply an extra SMTP header:
X-Verify: http://www.isp.com/verify.php?messageid@isp.com
Accessing the URL would provide a verification from the ISP that the message ID is valid and that the sender of the message properly authenticated with the server before sending the message. It could also return the authenticated user’s user ID, and an MD-5 hash of the message (to make sure that the message ID mathces the same message).
Then, you could filter your messages based on whether the messages were verified by a trusted ISP.
I’m not sure this would eliminate spam in itself, but it would certainly enhance Prof. Lessig’s bounty proposal by vastly improving the traceability of email.
Tom Brown
… one more comment: this would allow users to remain anonymous if they choose (they would choose not to have their emails verified), with the risk that their emails would be classified as spam. So long as there’s no gov’t proposal to REQUIRE verification, and so long as there’s no RBLing (which I too oppose) I think it wouldn’t pose too much of a threat to anonymous communication to those who wish to maintain anonymity.
It’s an interesting idea… but I don’t understand how it will reduce spam.
I think there are some underlying assumptions:
* the sender’s account with an ISP is the same as their email address (problem: uses user@attbi.com for DSL Service + relaying and myname@mydomain.com for email).
* most spam is not sent through “good” ISPs SMTP servers, so this doesn’t help in that sense.
* the solution requires changes to mail clients and mail servers
* the solution creates a dependency between SMTP and HTTP
* requires some kind of authentication to be used between the sender and the SMTP relayer
* who verifies the ISPs as being “good”? Some bright spark will decide to make a list of “good” ISPs and we’ll have blacklists en-masse again.
* If I understand the suggestion correctly, I could circumvent the whole apparatus by simply getting a shell account on some box and running my own SMTP service on it along with a “bogus” verification service.
* I’ve been emailing over the past few months with several sysadmins who have 20,000 users that they run mail for. The _insist_ that DNS blacklists are the _only_ solution that is computationally feasible for them to implement. To verify each incoming message, there’s quite a bit of extra work including an MD5 verification, DNS lookups, verification parsing and retrieval. And then when the sending SMTP service dies after the message has been delivered, there’s a whole failure routine that has to take place to deal with network inaccessibility, system failures, etc.
* Not to say that any of this is impossible, it ‘s just architecturally difficult to implement. Consider the example of a cluster of mail relayers that any large ISP will use. smtp.myisp.net might resolve round-robin against 5 physical machines (or even might sit behind a load balancer). All of a sudden, each of these machines needs to transmit message verification data to another web service that will be contacted by the receiving mail client to verify the message.
Is it possible to account for these questions in your idea, somehow?
I think I can handle most of those:
1) It’s fine for you to be you@domain.com and still use your ISP’s SMTP server (or, indeed, another SMTP server that you have an account on). The SMTP server simply verifies that you have an account on that server (i.e., it verifies that you are a subscriber to the ISP) – and gives your login name. It doesn’t verify the ‘from’ address.
2) True, but the “big” ISPs could offer this verification service, and we could presume that the big ISPs are “good.” We could also unbundle SMTP service from your ISP, and have separate companies offer verified SMTP sending. If the recipient trusts those companies, then he will not filter-to-trash messages verified by them. That way you’d still be free to choose a no-name ISP (though you may have to pay for the verified server service).
3) Only if you want to verify that a message is not spam. Then, yes. Or some other protocol if you prefer. Checking the verification header is optional and is performed by the recipient.
4) Why? Only the initial SMTP host needs to add the X-Verify header. It can verify the user’s identity through standard SMTP authentication.
5) I had envisioned each user making the choice for him/herself, and that the list would really be just some top ISPs and/or independent services. I had hoped it could be done without mega-blackhole lists. Perhaps not.
6) I don’t think so. What verification line would your bogus service add? It would have to be X-Verify: http://www.bogus.com/verify.php?messageid. But I don’t trust bogus.com as a verifier. I only trust isp.com.
7) This may be true. It may be computationally infeasible to calculate and transmit MD5 hashes for every message that comes through your doors.
I’ve long liked the “registered mail” idea on several fronts. Tom’s suggestion, making it part of a Lessig/Lofgren solution, is a particularly good application of it. (Presuming it’s the Tom Brown I think it is, we claim him around here as one of our best and brightest law/tech guys!) I also have long thought that law firms and other professionals would use and in fact pay a premium for a registered e-mail service, too, if it became trustworthy enough.
Let’s assume that this registered mail service is a good idea… and as Tom described it, I think there is some merit to it and perhaps I’m just hashing around implementation details.
To me, the “ultimate” solution is for everybody to use PGP/GPG/SMIME to sign their messages. That way, I can reliably create whitelists or block unwanted senders from landing in my mailbox. Server admins could also flat out reject mail that wasn’t signed by an acceptable signer (this is similar to how certificate databases in browsers work).
The major pitfall against this argument is that it becomes logistically complicated to roll out a huge PKI system to “the world”. But, technically, it seems like a highly attractive solution.
Maybe this registered mail service that Tom outlined is “easier” to implement (or maybe not?), but does it create a reliable platform for mail?
We’re getting drowned in blogland here. Googlesearch for ‘whirlycott’ and you’ll find my contact info easily enough… if you want.
Thank you for the info. http://www.bignews.com