Re: Server Side filtering (as pertains to Google Groups)

On 2008-08-15, VanguardLH <V@xxxxxxxxx> wrote:
Whiskers wrote:

[...]

There are NNTP commands which allow a client (including a local proxy
server) to save bandwidth by only downloading the headers required for
local scoring or filtering, including headers which are not included in
the overview.

HEAD Return the headers of an article (all the headers).

XHDR Return a certain header for a range of articles in a group.

XPAT Return a range of headers for articles matched by a certain pattern.

For the XPAT command (on some of the NNTP hosts where I tried it), some
error got returned about not supporting that command, even on the same
NNTP host which honored the HEAD command.

That's not unusual; XPAT puts a load on the server's resources, so NSPs
understandably don't all support it.

As a quick test, I ran:

telnet news.microsoft.com 119 (anonymous login)
help (check if head was listed)
group microsoft.public.windowsxp.newusers (as a test)
head (using 1st post in list)

Sure enough, all the headers got listed. Now to figure out if I can get
my NNTP client (40tude Dialog) to use the command so I can runs its
filters when retrieving the headers (to loop through them all again to
run 'head' and [re]apply filters). This will probably get me stuck
writing up another script for that client.

Have fun :))

Can leafnode or hamster be configured to retrieve all messages but just
their XOVER headers, as in a normal query (so not to consume my monthly
bandwidth)

Yes.

and then run 'head' against all of those message-IDs (in the
subscribed groups) to run its filters?

Not directly. The software would interpret the entries in the config and
filter files and then work out what NNTP commands and local processing to
use to achieve the desired results. One normally only uses the NNTP
commands directly when logged into a news-server using telnet or SSH.

I haven't used Hamster for years so I don't know how it reacts to a filter
that uses a non-overview header. I currently use Leafnode 2 but I have
nothing in the filterfile that requires use of non-overview headers so I
don't know how it would deal with such a filter. The mailing-lists or
user forums relating to those programs are probably the best place to get
definitive answers to such questions.

Leafnode 2 certainly supports HEAD XPAT and XHDR for local clients.

In principle is should be possible to write a proxy server (or newsreader,
of course) that uses the HEAD command for each article in the overview
[which has passed the filters which only rely on overview headers] and
then tries the headers against the filterfile before sending the BODY or
ARTICLE command for those articles which get past the filters. Whether
that would be quicker than fetching all the articles and then filtering, is
another matter - for most usenetters on a dial-up, the time on line is more
likely to cost money than the volume of data as such.

I've never used a news-server with a volume limit, but I imagine they
would be servers which carry a lot of binary stuff and the limits wouldn't
be significant for users interested only in text-only groups.

Supposedly my NSP doesn't charge
bandwidth on header-only downloads but I'm not sure running 'head'
(versus just 'xover') would qualify as a header-only download.

Your NSP is the best resource for answering such questions.

--
-- ^^^^^^^^^^
-- Whiskers
-- ~~~~~~~~~~
.

Relevant Pages

  • Re: [Dialog] Option to retrieve bodies of new posts does not work reliably
    ... I enabled the option "Retrieve bodies for all new posts". ... after retrieving their headers, there are still many posts that are ... But those filters will fail (and ... download the bodies of all new posts. ...
    (news.software.readers)
  • Re: Thoth "unsupported header" field
    ... Comcast uses Giganews and they don't provide all the headers. ... Delete those filters. ... Giganews doesn't *index* most headers in their overview.fmt database. ... And Supernews happens to be a bit cheaper than Giganews, ...
    (comp.sys.mac.comm)
  • Re: Can we exchange ideas for Thunderbird message filters?
    ... time ignoring headers afterwards. ... I don't think Thunderbird filters are working at all ... because it won't download more than 5000 posts in a group. ... P.S. Apologies for posting this in the dreaded GG too, ...
    (sci.math)
  • Re: Smarthost
    ... I'm not going to say that there are not filters out there that are picky ... about headers. ... headers, one for the original posters sending server, ... and finally one for your receiving server. ...
    (comp.mail.sendmail)
  • Re: what is wrong with google groups
    ... and the filters are removing about "30" spam keywords. ... OE is set up with headers in the right column, ... (currenly it's automatically removing about spam keywords). ... it for a dozen years before switching to Google. ...
    (rec.games.pinball)
Loading