Re: 40tude Dialog score on NNTP-Posting-Host

On Wed, 17 Jun 2009 19:13:52 -0500, VanguardLH wrote:

H-Man wrote:

On Tue, 16 Jun 2009 16:07:54 -0500, VanguardLH wrote:

H-Man wrote:

Can someone help with some indication as to how to score based on the
content of the mentioned header?

TIA

As I recall, the NNTP-Posting-Host header is not included in the
overview headers. That means you need to get more of the articles than
just their overview headers (to populate the headers list pane). That
means you have to download the articles (i.e., retrieve their bodies).
By retrieving the bodies of the articles, you also retrieve all of their
headers so you can then test on those non-overview headers.

Use the "Group -> Default group options" menu, Retrieving category, and
enable the "Retrieve bodies for all new messages".

That means you are retrieving not just the headers but also the bodies
of the articles. If you visit text-only newsgroups, the posts are
probably small enough that downloading them won't take a long time but
it WILL take longer. When you first subscribe to a newsgroup, there
could be many thousands of articles in it. Downloading all the bodies
(to get all headers) for all those posts will take time. Don't even
think about enabling the above option if you visit binary groups since
they are huge and you'll waiting an eon before all posts in a large and
busy binary group get downloaded in their entirety.

I have the above option enabled. I visit text-only newsgroups. Because
the entire article gets downloaded, I can test on the NNTP-Posting-Host,
Path, and other non-overview headers. Dialog actually makes 2 passes
through its filters (which can produce side effects if you're not
careful in defining your filters). The first pass is made when Dialog
downloads only the headers of the articles. Your filters are exercised
against this list. That means any filters that test against the
overview headers get exercised in this first pass. Then the bodies are
downloaded and the filters exercised a 2nd time. What you might see are
articles that are not downloaded in the 2nd pass to get the filters
exercised against them again. For example, say you have a filter that
ignores articles from Google Groupers (i.e., a filter looks at the
Message-ID header for googlegroups.com). In the 1st pass, the filter
has tagged all Google Groups posts as ignored. Ignored posts don't get
downloaded in the 2nd pass. The result is that you will see some posts
that are ignored and also not downloaded for their bodies in the header
list pane (there is no "page" icon next to the article in the header
list to show the article got downloaded - but it didn't get downloaded
after it was deemed an ignored article).

Downloading the bodies of the posts takes more time even in text-only
newsgroups. If you can tolerate the extra delay, you get all the
headers on which you can then test. Many times the overview headers are
not nearly sufficient to define a kill filter. You need the
non-overview headers to create a kill filter that more accurately
targets the unwanted post or poster.

Thanks, this has been a very educational thread for me. Thanks also to Rom.
Another question is the by scoring on Message-ID (it's working for me for
now), why does scoring at +9999 show a collapsed thread as that score but a
-9999 shows the thread as unscored until the thread is expanded? It's not a
big deal, just wondering why, it seems rather inconsistent to me. I guess
I'd like to see the thread started by a doofuss as started by a doofuss
rather than expanding it first to see it's a waste of time.

Sorry but I've never bothered with scoring of posts. Rather than
accumulating a score on a post to grade its spamminess (or some other
unwanted trait), I just use actions in Dialog's filters. Once the
filter identifies a matching post or poster, I'm not interested in
combining other filters to qualify the first filter. For example, to
get rid of Google Groupers (which is a lot of spam), I use:

[*]
...
# Dim posts through Google Groups.
!setcolor(gray;white) Message-ID {<.+\.googlegroups\.com>}
...
[* -{^microsoft\.(public|private)\..*}]
...
# Ignore posts through Google Groups.
!setcolor(silver;white),ignore,markread Message-ID {<.+\.googlegroups\.com>}

This colorizes (dims) posts by Google Groupers in all groups; however, I
later check if the Google Grouper is *not* posting in a Microsoft
newsgroup and, if so, ignore those posts (and use a view that hides
ignored posts). A lot of spam is spewed out through Google Groups
(because accounts there are free and require no real validation of the
user that opens an account with them). I don't want to see any Google
Groups spam in any newsgroups that I visit EXCEPT for Google Groupers in
the microsoft.* newsgroups. There isn't anywhere as much Google Groups
spam posts showing up in the Microsoft groups as there is in the
non-Microsoft groups.

I ignore instead of delete. An option in Dialog lets me apply the
watch/ignore tags to an entire subthread. I don't want to see the
unwanted post but I also do not want to see any replies to it. If I
delete the post, there is no subthread to keep track of so I would see
replies to it. By ignoring the unwanted post (which also ignores all
replies to it) and hiding it, any replies will also get ignored and thus
hidden in the Hide Ignored Posts view that I defined. Also, if someone
makes mention of a post that I've ignored, I can switch to the All
Messages view to go look at what they are talking about.

There is no point in scoring posts by these unwanted posters. I know
that I don't want to see them. I don't need to accumulate a score to
reach some threashold at which I hide them. Using a high score value
alone as a threshold doesn't make any sense because I can do the same
with just using actions on the post.

Many thanks, this was very helpful.
--
HK
.

Relevant Pages

  • Re: Pytanko do sciepowych kapusiow... PS
    ... suspect you may have run afoul of our abuse filters, ... This address is only for reporting spam or abusive posts originating ... be sure to include the entire headers from at least ...
    (soc.culture.polish)
  • Re: [Dialog] Option to retrieve bodies of new posts does not work reliably
    ... I enabled the option "Retrieve bodies for all new posts". ... after retrieving their headers, there are still many posts that are ... But those filters will fail (and ... download the bodies of all new posts. ...
    (news.software.readers)
  • Re: 40tude Dialog score on NNTP-Posting-Host
    ... just their overview headers. ... the posts are ... Downloading all the bodies ... careful in defining your filters). ...
    (news.software.readers)
  • Re: Can we exchange ideas for Thunderbird message filters?
    ... time ignoring headers afterwards. ... I don't think Thunderbird filters are working at all ... because it won't download more than 5000 posts in a group. ... P.S. Apologies for posting this in the dreaded GG too, ...
    (sci.math)
  • Re: Conceal Identity
    ... How do I do a search on newsgroup headers? ... While you can control the name and email address, ... Your NNTP provider controls the rest. ... Are lines 4 to end unique to me and identify me in all my posts? ...
    (microsoft.public.windows.inetexplorer.ie6_outlookexpress)