Re: Google spidering & traffic

Here are spider bandwidth stats for the 19th

Googlebot (Google) 262.93 MB 19 Feb 2007 - 23:54
Inktomi Slurp 74.83 MB 19 Feb 2007 - 23:58
MSNBot 270.99 MB 19 Feb 2007 -
Unknown robot (identified by 'robot') 597.13 KB 19 Feb 2007 - 05:23

Thats 1.2+ Gig for the day?

Yet when I look at my overall summary for the 19th:

Day Number of visits Pages Hits Bandwidth
19 Feb 2007 287 24266 45669 181.29 MB

I smell a rat...


On Feb 20, 4:25 pm, "Jukka K. Korpela" <jkorp...@xxxxxxxxx> wrote:
Scripsit Cinamon Thunder:

Google and other engines spider my website every single day and as
result generates QUITE A BIT of traffic.

Do you mean the _spider visits_? Even if a spider visits your all pages
every day, that's still just one HEAD request per page. If that's
considerably more than normal traffic to your site, I'd be more worried
about the low usage than the spiders.

(I'm assuming that your server sends adequate Last-Modified headers so that
spiders can just send a HEAD request and see that nothing has changed,
instead of retrieving the actual content.)

<META NAME="revisit-after" CONTENT="7 days">

Important search engines probably ignore that routinely, since there has
been too much abuse.

Any advice would be appreciated!

Please convince me that daily visits by spiders are a _problem_. Many
webmasters would pay real money to achieve such a situation.

Jukka K. Korpela ("Yucca")