Re: Allow robot access to protected content

Thanks to all for the replies. I had no idea this was such a sensitive
issue. If our publication has information that could be helpful to
someone, I figured they should know about it. I guess that upsets some.

However, I'm pretty sure I've come across dozens of legitimate sites on
SE's, particularly on Google News, that require registration. Wall
Street Journal is one example that comes to mind. Their link shows up
with a "(subscription)" tag next to it, and I was just wondering how
they get that done.

I may be misunderstanding the terms and concepts here; perhaps Google
News is not strictly a search engine, and it is only there that it's
allowed.

(As an aside, re the cache issue, I was under the impression that a
"robots=nocache" meta tag prevents the search engine from showing a
cached page.)

.

Relevant Pages

  • Re: Can there be any doubt at this point that queries were a bad idea?
    ... Garrett Smith wrote: meta tag and be done with it. ... but it has nothing to do with search engine indexing. ... Also, even if it were, this would not help, because other sites over which I have no control can link to my pages which I don't want indexed. ... I believe that Google honours robots.txt, but as I said above, Google ignores: ...
    (comp.infosystems.www.authoring.html)
  • Limited number of dynamic pages indexed by spider
    ... We've built and launched a site that we thought was search engine ... Anyone have any idea how we can change portions of our URL or ... We have also set it so the product title (as entered by the seller) ... shows up in the title tag and as a portion of the description meta tag. ...
    (alt.internet.search-engines)
  • Limited number of dynamic pages indexed by spider
    ... We've built and launched a site that we thought was search engine ... Anyone have any idea how we can change portions of our URL or ... We have also set it so the product title (as entered by the seller) ... shows up in the title tag and as a portion of the description meta tag. ...
    (alt.internet.search-engines)
  • Re: Hiding a line of text from the SE crawler
    ... Actually a meta tag has nothing to do with a search engine "finding" a site. ... I want the spider to read>> and crawl all the other information on the page but not the navigation bar ... > Go join a seo group and find out more before you make this big mistake. ...
    (microsoft.public.frontpage.client)