Google Bots Crawling!

[edgarf76]

I am seeing record high levels that google bot is crawling. Are all of you guys experiencing this?

 

[TheGooner]

My site has approx 30,000 pages and I get between 4,000 and 10,000 pages crawled a day by googlebot. (Peaked at 12,000 in May).
Last few weeks have been at the low end (around 4,000 pages) but 28/3 was 6,260 pages.

So a slight wiggle but not anything major.

 

[Frank]

I am getting about 2000 to 4000 crawled on my main site, which has around 3800 pages, and between 150 to 500 on the new site which is under 3 months old but has around 300 pages, it looks normal to me so far.. its when i see a huge spike i worry...

 

[inspiration]

creepy crawlies go away LOL

 

[edgarf76]

Could the big spike be because I added a lot of Schema code over the past few months?

 

[RyanWeb]

Any update from anyone? I see some new terms my sites are starting to receive traffic for..

 

[Vivi Fox]

You can prevent Google bots from crawling content on your site by using the following methods...

1. Use a robots.txt robots exclusion file. ...

2. Use “noindex” page meta tags. ...

3. Password protect sensitive content. ...

4. Nofollow: tell search engines not to spider some or all links on a page. ...

5. Don't link to pages you want to keep out of search engines. ...

6. Use X-Robots-Tag in your http headers.