Re: Extended 5xx code (was Re: Drop UCE instead of forwarding off-site?)

"Xavier Roche" <xroche@xxxxxxxxxxxxxxxxxxxxxx> wrote in message
news:guoe0i$6b2$1@xxxxxxxxxxxxxxxxxxx
D. Stussy a écrit :
The proposal for "ExtendedErrorDrop" is so that a back-end server can
tell
a front-end that a drop is preferred to generating an NDR for CERTAIN
types
of rejections (based on the DSN returned, and ONLY those DSNs indicated
by
the proposed option).

Do the "race conditions" really exist in the real world ? Considering
the very few numbers of race conditions, the sane solution might be to:
(1) have an efficient filtering on all frontends with 5XX rejections
(2) for all backends, accept all mails from the frontends, without any
costly filtering at all

Note that you can also play with greylisting (4XX "try again later"
codes class) for suspicious mails, to ensure that the next try will have
a reliable RBL listing (good/evil) - this will probably increase the
efficiency much better than having multiple filterings.

E-mail is not instant messaging. In fact, a forwarding system can hold a
message for hours. All "reasonable" MTAs perform their checks up front but
that's it; they don't check again when sending.

Does the race condition exist?

Theoretically: Yes. Even a few microseconds is enough for a malicious
message-related database to be updated between checks by two MTAs in any
message's forwarding path.

In Practicallity? Just because it isn't observed doesn't mean that it will
never happen as long as it cannot be proven to be eliminated from ever
happening. Any proper design must account for the case where it does
happen, even if just once in a decade. The longer a message is in transit,
the higher the probability that it will span an update cycle of the
external service that could change the malicious status (from not-listed to
listed) of a message.

Probability? There are two factors:
a) The probability of a forwarded message being forwarded across update
cycles. How often with two checks by MTAs fall on opposite sides of update
cycles? This can be calculated as update rates and durations, plus average
forwarding time are all measurable.

b) The probability that a given source will update for a particular
message. Over enough time for any given message sent to thousands or
millions of recipients, this should approach 1 (and be so close, we may as
use 1). However, if our copy of the message is among the first few copies
sent out, the probability will be or be close to zero. Obviously, the
probability of a service to have a message (or its source, for DNSBLs)
listed increases from zero to one as more copies are sent. It will
eventually hit a system (or enough systems) to cause a listing somewhere.
For any given instance of a message, this probability is not calculable by
its recipient, but that doesn't mean it should be ignored. The party
capable of computing the probability is the spammer! He will know how many
recipients have received the spam and perhaps know how often those
recipients report spam.

The overall probability of the race condition occuring for any service is
the product of the probabilities under factors A and B. It is an
expression of occurance over time. When considering multiple services, one
needs to take the union of probabilities less their even-numbered
intersections, plus odd-numbered intersections: Mathematically, the sum of
their individual probabilities less the sum of their products of
even-numbered combinations, plus the sum of products of odd-numbered
combinations (as we only want to count overlaps once, not twice or more).

Border conditions:
Zero: If a given message has not been seen before, is not similar to any
other spammy message, and your system is the first MTA to see it, the
probability of a race condition occuring while your front-end is forwarding
starts at zero (and increases from there over time as more recipients
receive it).
One: Unless all sources used are under continuous update, oscillating on
the spam status for a given message, and you are testing with respect to
that message, a probability of one should not occur. However, for a
sufficiently old spam message, a probability of one can be approached.

Will the race condition ever happen? As long as it has a non-zero
probability, it cannot be eliminated. Therefore, it will happen sometime
to someone. Do you want to be that someone who gets listed on some
anti-backscatter list because you hit the race condition and your front-end
MTA backscattered?


Having the back-end (forward recipient) MTAs whitelist their front-ends
without any malicious message checks is not responsible. Although it also
would avoid backscatter, it lets more spam in than if the back-end
re-checks everything for itself. (Some DNSBL checks are the only ones
skippable, since such messages would be testing the IP address(es) of the
front-end, not the message source.)

The above assumed front and back end servers under a unified or
sufficiently cooperative control. There are also forwarding situations
(e.g. registrars with private "whois" services, forwarding and MX backup
services, etc.) where the forwarder and recipient are under separate
adminstrative control and/or don't even use the same malicious message
detection services. The proposal is a valid solution for their interaction
as well.

I do recognize the merit of greylisting as a way to delay the malicious
status check to give the classification services time to generate a
listing. However, I note a problem with it: If EVERYONE used it, the
situation would be no different than if no one used it, except that spam
which is tried only once would NEVER reach a classifying service. Retried
spam would be listed with a delay. Personally, I don't use it because I do
not believe in hindering the flow of legitimate messages in any [avoidable]
way, while GL will delay delivery of legitimate mail (even if just a few
minutes).


.

Relevant Pages

  • Re: Can anyone explain Bayesian filtering/calculations?
    ... > moment is to use Bayes' Rule to handle the probability calcs... ... If we take spam as our text: any word might appear in spam, ... word might appear in not-spam (ham). ... int main ...
    (comp.programming)
  • bayesian spam filter
    ... I have been looking at bayesian based spam filters and have a question about ... where we read this as the probability of the message, containing the word w, ... of choosing that word out of the entire corpus of known words. ... are spam, and 5,000 emails you know are not-spam and you use this to train ...
    (sci.math)
  • address harvesting (was: For sale: HP Integrity rx2600 parts (various))
    ... anyone harvests usenet for email addresses. ... that they respond positively to spam is much lower than in the general ... I get typically a one digit or low two digit number of ... Real names enhance the probability of getting real answers. ...
    (comp.os.vms)
  • Re: combining probabilities
    ... down pat but I'm having trouble understanding the following problem. ... If word A indicates an e-mail is 90% likely to be spam and word B indicates ... derivation or proof to full understand how this combined probability works. ... You'd like the conditional probability Pthat the email ...
    (sci.math)