Re: archival/compliance/etc.

On Wed, 19 Jul 2006 09:43:03 -0700, Robert Harker wrote:

Hi Robert - I took your course in Chicago a couple of years ago. Good
stuff.



Kevin K wrote:
Currently I use an open-source program called mairix which indexes bodies
of e-mail for searching on sender, recipient, date, subject, body, etc.

It works but isn't scaling well to the volume of mail we handle.

I think a different (better) solution is to reduce the number of
messages that you are archiving. Do you archive all your mail?
If so, do you really need to? If you can define the users who need
to have their mail archived then you can add glue to only archive
those messages.


We're required by SEC to archive mail to/from certain users, and our legal
department has given us a policy regarding archiving mail to/from all
other users.

I've been told that we can omit mail flagged as junk by our anti-spam apps
(Brightmail and Barracuda), which helps a lot.



One way to do this checking would be to have sendmail do the checking.
Create a hash database with a list of addresses that need archiving.
Test both the sender and the recipient address against the database
in the check_mail and check_rcpt rulesets. If a match is found, set a
macro which is then passed to a Milter that would add an archive Bcc:
address if the macro had been set. MIMEDefang would be a good choice.

I'm playing with MIMEDefang now and might use it. Right now logall.c lets
me avoid archiving the junk and doesn't use much in the way of system
resources.



Another issue you brought up was capturing Bcc: addresses. First off
in terms of SMTP there is no such thing as a "Bcc:" address. The Bcc:
header is a way for a sender to add additional envelope recipient
addresses to a message that do not show up in the message itself.
It is normally used by the user's MUA (Mail User Agent). When the
MUA passes the message to the MTA (Mail Transport Agent) it will
pass all the recipients in the To:, Cc: and Bcc: headers as envelope
recipient addresses, but while it will generate To: and Cc: headers,
it does not generate a Bcc: header so these recipients do not show
up in the message itself.

Right ... so far it's worked okay as long as I can search for an address
anywhere in the header. But it's not perfect.

I guess to perfectly capture bcc:d mail, the archiving would have to be
integrated into the mail client.


A second issue is forwarding. Is the message forwarded to an internal
relay that splits internal and external mail. I.e. internal mail
to the mailbox server and external mail to the firewall SMTP relay.
If you are trying to capture the Bcc: addresses on the firewall
SMTP relay, you will miss any internal Bcc: addresses. So the most
reliable place to capture the Bcc: addresses is on the first SMTP
host that receives the message.


Right now I run sendmail with logall.c at each step (individual internal
relays and external gateways) and have logall ignore mails that have
already been tagged as logged, to avoid duplicate archival.



BTW, you can also derive Bcc: addresses from the syslog files if you
also log the To: and Cc: headers. This could be done with a pair of
header specific rulesets for To: and Cc: headers and the sendmail's
syslog database.

Hmm. That might be useful.


Now what to do with these Bcc: addresses? You could have MIMEDefang
log them. You could also have MIMEDefang generate an X-Bcc: header
and list them in the message header. Simply adding an X-Bcc: header
may not be the best idea since it would expose the Bcc: addresses to
all of the recipients, not just the archive address.

The best way to do this would be to make it a conditional header
(H?F?X-Bcc: or H?{Macro}?X-Bcc:). I don't think H?{Macro}?X-Bcc: would
work since Milter does not have any mechanism to pass a macro back
to sendmail (is this correct?) I also do not know if Milter allows a
header to be returned with ?F?X-Bcc: conditional mailer flag syntax.
If it were allowed, you could then use an unused mailer flag and then
define a custom mailer that sets that flag. you would then deliver
mail to the achieve address with this mailer using the mailertable.

Not sure if this would work or not, but these are my thoughts.

Hope this helps


Lots of good ideas, Robert. Thanks very much.



.

Relevant Pages

  • Re: archival/compliance/etc.
    ... of e-mail for searching on sender, recipient, date, subject, body, etc. ... Create a hash database with a list of addresses that need archiving. ... add an archive Bcc: ... it does not generate a Bcc: header so these recipients do not show ...
    (comp.mail.sendmail)
  • Re: BCC Question
    ... No recipients have to be in the header for a message to be delivered; ... and leaves the recipient fields in the headers blank or enters ... Quick question regarding BCC. ... this from the header before delivery? ...
    (microsoft.public.exchange.admin)
  • Re: Receiving email meant for others
    ... You should not see any recipient in BCC, ... CamD wrote: ... > that header. ...
    (microsoft.public.exchange.admin)
  • Re: Receiving email meant for others
    ... I should see the BCC in the headers. ... This is the MIME raw ... header. ... >You should not see any recipient in BCC, ...
    (microsoft.public.exchange.admin)
  • Re: X-No-Archive, here we go again ;-)
    ... Only posted to those who reply to me using the X-No-Archive header. ... it used to be that the XNA ... flag prevented one's post from being seen on Google ... archiving database, it just adds to the noise. ...
    (news.software.readers)