Re: reply urgent
- From: John Bokma <john@xxxxxxxxxxxxxxx>
- Date: 20 Feb 2008 23:05:16 GMT
CreativeMind <aftab.pucit@xxxxxxxxx> wrote:
google spiders all pages of a website?? actually i use some pages for
just returning values or some specific purpose.They are not intended
to be spidered.
The best option is to password protect such pages. There are plenty of
"spiders" that don't check robots.txt
--
John Bokma http://johnbokma.com/
.
- References:
- reply urgent
- From: CreativeMind
- reply urgent
- Prev by Date: Re: reply urgent
- Next by Date: Re: blocking robots.txt from non-robots
- Previous by thread: Re: reply urgent
- Next by thread: Re: reply urgent
- Index(es):