Re: Allow robot access to protected content



Thanks to all for the replies. I had no idea this was such a sensitive
issue. If our publication has information that could be helpful to
someone, I figured they should know about it. I guess that upsets some.

However, I'm pretty sure I've come across dozens of legitimate sites on
SE's, particularly on Google News, that require registration. Wall
Street Journal is one example that comes to mind. Their link shows up
with a "(subscription)" tag next to it, and I was just wondering how
they get that done.

I may be misunderstanding the terms and concepts here; perhaps Google
News is not strictly a search engine, and it is only there that it's
allowed.

(As an aside, re the cache issue, I was under the impression that a
"robots=nocache" meta tag prevents the search engine from showing a
cached page.)

.