Separate Servers for Humans vs. Bots with Same Content Considered Cloaking?

[Desiree-CP]

Hi,

We are considering using separate servers for when a Bot vs. a Human lands on our site to prevent overloading our servers. Just wondering if this is considered cloaking if the content remains exactly the same to both the Bot & Human, but on different servers.

And if this isn't considered cloaking, will this affect the way our site is crawled? Or hurt rankings?

Thanks

[ThompsonPaul]

The additional massive complexity, expense, upkeep and risk of trying to run a separate server just for bots is nowhere near worth it, in my opinion. (Don't forget, you'd also have to build a system to replicate the content between each server every time content/code is added or edited. That replication process could well use more resources than the bots do!)

I'd say you'd be much better off using all those resources towards a more robust primary server and let it do it's job.

In addition, as Lesley says, you can tune GoogleBot, and can actually schedule Bing's crawl times in their Webmaster Tools. Though for me, I'd want the search engine bots to get in and index my site just as soon as they were willing.

Lastly, it's only a few minutes' work to source a ready-made blacklist of "bad bots" useragents that you can quickly insert into your htaccess file to completely block a significant number of the most wasteful and unnecessary bots. You will want to update such a blacklist every few months as the worst offenders regularly change useragents to avoid just such blacklisting.

Does that make sense as an alternative?

Paul

[LesleyPaone]

I second what Jonathan says, but I would also like to add a couple of things. One thing I would keep in mind is reserve power on your server. If you are running the server close enough to its maximum traffic limit where a bot would matter, I would upgrade the whole server. All it takes is one nice spike from somewhere like hacker news or reddit to take your site offline, especially if you are running close to the red.

From my understanding you can actually adjust how and when Google will crawl you site also, https://developers.google.com/search-appliance/documentation/50/help_mini/crawl_fullcrawlsched

[Jonathan1979]

I've never known search engine bots to be particularly troublesome and overload servers. However, there are a few things you could do:

1.  Setup Caching

2.  Setup something like Cloudflare which would be able to block other threats.

I cannot imagine you are intending to block google, bing etc as I would definitely advise against cloaking the site like that from Google.

Of course it is difficult to make any specific comment as I have no idea to the extent of the problem you are suffering from.  But something like caching \ cloudflare security features will help alot.