Why SMTP Sucks...mh003

Why SMTP Sucks
--------------

This document is an outline of why spam is more of a problem in SMTP
than in any other protocol.

For the purpose of this document, we define "spam" to be "messages
which the recipient determines to be spam."

The document is organized as follows. The first section describes
the features of SMTP which enable spam. The next section compares
other protocols to SMTP, and shows why spam in those protocols is not
as much of a problem as it is in SMTP. The last section describes
changes we can make to the use and deployment of SMTP, which may help
address the spam problem.

----------------------------------------------------------------------

Features of SMTP which enable spam
----------------------------------

The following is a list of features in SMTP which enable spam.
Other protocols have some, but not all of these features. It is our
contention that the *combination* of these features results in the
spam problem being worse in SMTP than in any other protocol.

Note also that we're not saying that these features are *bad*. They
may be necessary for the design of any similar message delivery
protocol. Out position is again that the *combination* of these
features, and the *extent* to which they're (ab)used may not be
required for a message delivery protocol.


--

1. SMTP is any to any

Any sender can connect to any SMTP recipient, and exchange traffic.

e.g. POP doesn't have this problem, because it's any *known-user*
connecting to a *particular-server*. There's no incentive to hammer
at POP servers (except for buffer overflows, etc.), because any
traffic will never get through.

e.g. HTTP does have this feature. But as we will see later, it
doesn't share other features of SMTP, so spam in HTTP isn't as bad as
spam in SMTP.


2. Senders can claim to be anyone

Senders can forge messages (envelope and body) from anywhere, as
anyone. This means you have no idea if the person claiming to send a
message is really who they claim they are, or if they're associated
with who the claim.

Spammers claim association with pre-existing domains because such
fraudulent associations are useful for criminal purposes.
e.g. Account scamming, etc.

e.g. IRC doesn't have this feature, because the users are centrally
authenticated. This means that they can be "kicked" or "banned" from
the network.

e.g. HTTP doesn't have this problem, because for 99.9% of HTTP
traffic, the user isn't authenticated, and the server is sending
well-known pages to anonymous recipients who've requested them.

e.g. SMS or text paging doesn't have this problem, because the
originator is well known.


3. The sender can request that the message be forwarded to (almost)
anyone else.

SMTP isn't a "user to user" protocol. It's a "submitter to MTA"
protocol. In the case of open proxies, the MTA may be abused to
re-send the mail to anyone on the net.

In the case of MTA's which are not open proxies, the sender can
still request that the MTA forward the traffic to any one of a number
of users who deals with that MTA.

e.g. NNTP doesn't have this problem, because a message exists for
the entire server, not for a particular user on that server. Each
user maintains their own individual list of read & unread messages,
which relieves the server of that burden.

e.g. POP doesn't have this problem, because the only messages in the
visible POP store are for the current authenticated user.

e.g. SMS or text paging doesn't have this feature, because it's a
one to one delievery system. Cell phones don't forward messages,
unless specifically instructed to do so by the owner.

Additionally, this means that "dictionary attacks" become feasible
in SMTP. The spammer can try reasonable user names, and cause
messages to be forwarded to that user even if they don't know whether
or not the user exists.


4. Data flow in SMTP is biased towards originators.

A sender can supply large amounts of data to an SMTP server, and the
server is obliged to accept it. This is a desired feature for a
message delivery protocol, but it's being abused by spammers.

e.g. HTTP is biased towards senders. "GET /", returns a 10K page,
so it's biased towards servers sending traffic, and HTTP "originators"
receiving it.

This bias in HTTP is abused by domains which have huge front pages.
The average user hits the "stop" button on their browser if the page
takes too long to load. This indicates that the users don't
appreciate being abused in this manner.

e.g. Any message delivery protocol has this problem.

e.g. POP has this problem, but the messages are delivered to the
recipient only when the recipient requests them to be delivered. The
recipient can also obtain some information about the message prior to
receiving it, and can delete the message un-read.


5. The sender can amplify his traffic

e.g. Multiple RCPT TO's. They're nice for people you know
delivering email, but why are people *not* known to you trying 15 RCPT
TO's? The answer is that they're trying to lower their costs for
sending spam...

This behaviour is even worse when open proxies get involved. A
spammer can use a 56k dial-up to send messages to a large MTA, and
have it flood a T1 with outgoing traffic.

e.g. NNTP. One message gets delivered to all cooperating servers.
It's a flood fill of the network, but to *servers*, not to *users*.
MTA's currently do something similar when they use multiple RCPT TO's
when forwarding messages. The messages are grouped by recipient
domain, and only one copy is delivered.

But this method doesn't work when the MTA has to store the messages
locally. It often stores one copy of the message in each of many
users mailboxes.

e.g. ping floods have this problem (I forget what they're called...)
forged ICMP echo requests from a victim IP, to a network. All of the
machines in the network respond, and flood the victim. This attack,
however, can be trivially stopped with firewall rules on the abused
network: don't allow packets to be sent from outside to the network
address.


6. no network-based negative feedback

TCP has congestion control. ICMP "port unreachable", etc.

When SMTP messages are thrown away, they're often done so by the end
user. The recipient MTA usually doesn't know, and the originating MTA
doesn't know. Also, the originating MTA isn't "punished" for sending
the garbage traffic, or told to self-limit the rate of messages.

So in the absence of negative feedback, spammers increase their
sending rates, in the hope that some messages will get through. The
recipient MTA's do nothing, and spammers have every incentive to
participate in, and create, what is effectively a positive feedback
system.

----------------------------------------------------------------------
Other Protocols
---------------

TBD. Go through each protocol (NNTP, HTTP, IRC, RSS, "pull" email,
etc.). For each one, explain why it does (or does not) have the
features listed above. Describe how much of a problem spam is for
that protocol, and how it has been addressed.

----------------------------------------------------------------------

What to do about it
-------------------

All of these proposals are "off the cuff", and deserve more detailed
examination.

Also, many people have a knee-jerk reaction that any solution is of
the "either-or" style. Either you apply all of the solution, or you
apply none of it. This attitude is irresponsible.

What we're proposing is a "grab-bag" of solutions. An MTA may apply
all, some, or none of these solutions to all, some, or none of the
messages it receives. That is, the behaviour of the MTA should be
adaptable, and should vary depending on the information available
about a message.

e.g. Messages which get delivered to a corporate head office from
satellite groups may be passed through virus filters, but may not
get passed through LMAP, or other content filtering schemes.

e.g. A message from a partner company with a well-known IP may be
passed through virus filters, but may not get passed through LMAP,
and any content filters may err on the side of caution.

e.g. Messages from unknown senders should be treated with great
suspicion. Any and all available information should be used to
determine how to process the message.


Dealing with SMTP's Features
----------------------------

1. Removing the "any to any". e.g. LMAP.

This has the problem that people claim SMTP is a "end to end"
protocol. It's not. SUBMIT has been around for a while. POP before
SMTP has been around for a while. Users shouldn't be sending mail via
the MTA to MTA delivery protocol. The design requirements of the
protocols are very different.


2. Removing forgery. e.g. LMAP, Yahoo's "DK" proposal to sign
messages, and to put keys into DNS.


3. If a particular sender tries to send messages to some number of
users who don't exist, then that sender should be told to back off,
and further messages should be treated with great suspicion.


4. SMTP is about message delivery. We can't do much about senders
delivering messages.

The concept of "pull" email is often proposed to deal with this
issue. We're not convinced that the pull ideas are any better than
SMTP, At the minimum, they have to address the other features of SMTP,
too.

Maybe an updated consent protocol. Senders can also publish
information about their intentions, in the SMTP protocol.

Sender: "I've got 10 messages to 5 recipients, with a total data size
of 1M."
Recipient: "I'm busy, so send me only 2 messages, up to 200K".
Sender: OK, here they are.


This means that the sender has to make his intentions clear. If his
actions disagree with his stated intent, then the recipient can drop
the connection, and trated treat the delivered messages with
suspicion


5. Don't allow multiple RCPT TO's for unknown senders.

The problem is that you can't selectively NACK later RCTP TO's. The
sending MTA then decides that the entire message was NACK'd, and
discards (or retries) it.

Q: Is this a feature requirement of SMTP, or is it just something
that the authors of MTA's decided to do, because it seemed
reasonable?

The idea behind disallowing multiple RCPT TO's is not to stop the
traffic. You already KNOW that spammers are sending multiple messages
with the same content. The intent is to force the spammers to send
more traffic (yes, you have to deal with it, too), but as a
side-effect, the amount of traffic from any one spammer is reduced.

If the spammer decides to get around the multiple RCPT TO's by
opening more connections from the same IP (or even different ones),
then Greylisting may be used to block their messages.


----------------------------------------------------------------------

Ramblings
---------

If we patch smtp, then we will have the problem of deployment. Old
machines will not *ever* be upgraded, so we will have to accept
traffic from them, and not apply the new rules. This may
significantly reduce the effectiveness of any new system, as it must
be backwards compatible with old (spam-ridden) systems.

So there is some incentive to move to a new protocol/port, even if the
new protocol is SMTP + hacks. If nothing else, it gets rid of old
machines which haven't been upgraded, (and moves to a new set of
problems.) But if the architecture of the new system provides all of
the positive features we wanted from SMTP, with few of the negative
features which enable spam, then the problems will at least be fewer
than in SMTP.

----------------------------------------------------------------------

Recommendations
---------------

- allow for the selective NAK of multiple RCPT TO's.

- Some kind of qualification of MTA's: LMAP, greylisting, etc.

- senders should publish their intentions, rather than forcing data
down the recipients throat.

Alan DeKok.


.