Aussie-Dave
Former AGD Member
- Joined
- Nov 24, 2007
- Messages
- 684
- Reaction score
- 3
Hi All,
baidu.com is literally sucking bandwidth on a multiple daily basis.
It's not uncommon to see it come in pull the index page not once but 5 times in less than 1 hour. It doesn't follow robots.txt instructions - even though it does access the file.
It's totally out of control...
The problem with baidu bot it comes through on different C-Classes so blocking it becomes nightmare.
There are a few other bots like this that do the same.
exabot.com
sos.com
Anyway, just a heads up to check for these and other bots that not only inflate stats but pig out on your bandwidth for no reason at all.
Cheers
Dave
baidu.com is literally sucking bandwidth on a multiple daily basis.
It's not uncommon to see it come in pull the index page not once but 5 times in less than 1 hour. It doesn't follow robots.txt instructions - even though it does access the file.
It's totally out of control...
The problem with baidu bot it comes through on different C-Classes so blocking it becomes nightmare.
There are a few other bots like this that do the same.
exabot.com
sos.com
Anyway, just a heads up to check for these and other bots that not only inflate stats but pig out on your bandwidth for no reason at all.
Cheers
Dave