Controlling crawl speed/delay through dynamic server-code and 503's
-
Lately i'm experiencing performance trouble caused by bot traffic. Although Googlebot is not the worst (it's mainly bingbot and ahrefsbot), they cause heavy server load from time to time. We run a lot of sites on one server, so heavy traffic on one site impacts other site's performance.
Problem is that 1) I want a centrally managed solution for all sites (per site administration takes too much time), which 2) takes into account total server-load in stead of only 1 site's traffic and 3) controls overall bot-traffic in stead of controlling traffic for one bot. IMO user-traffic should always be prioritized higher than bot-traffic.
I tried "Crawl-delay:" in robots.txt, but Googlebot doesn't support that. Although my custom CMS system has a solution to centrally manage Robots.txt for all sites at once, it is read by bots per site and per bot, so it doesn't solve 2) and 3).
I also tried controlling crawl-speed through Google Webmaster Tools, which works, but again it only controls Googlebot (and not other bots) and is administered per site. No solution to all three of my problems.
Now i came up with a custom-coded solution to dynamically serve 503 http status codes to a certain portion of the bot traffic. What traffic-portion for which bots can be dynamically (runtime) calculated from total server load at that certain moment. So if a bot makes too much requests within a certain period (or whatever other coded rule i'll invent), some requests will be answered with a 503 while others will get content and a 200.
Remaining question is: Will dynamically serving 503's have a negative impact on SEO? OK, it will delay indexing speed/latency, but slow server-response-times do in fact have a negative impact on the ranking, which is even worse than indexing-latency.
I'm curious about your expert's opinions...
-
Hi INU,
I always like avoid using things like 503s as a general rule. There is almost certainly a better way to do it.
What about just using Google webmaster tools and Bing webmaster tools? Regarding HREFs it depends how much you rely on that tool. If you don't use it, then I'd more more likely to just block that bot in robots.txt and make sure Google and Bing are controlled using the appropriate tools in the respective webmaster tools.
To answer your specific point about whether or not 503 can hurt rankings. In general no as long as they are only short-term. A 503 like 404s or any other response code is a natural part of the web, however, Google has said in the past that repetitive 503s can be treated as permanent rather than temporary and in some cases can result in the pages being removed from the index.
I hope this helps,
Craig
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Penguin 2.1 Penalty- Can't Understand why hit by it?
Hi, I have lost all my rankings after Penguin 2.1 update. I haven't did anything wrong. Want to know the root cause of the penalty so that I can overcome this. Any help would be appreciated. Website: http://tiny.cc/hfom4w
White Hat / Black Hat SEO | | chandman0 -
Negative SEO on my website with paid +1's
Hi guys, I need a piece of advice. Some scumbag played me quite well with paid +1's on my two articles and now I'm in a problem.
White Hat / Black Hat SEO | | Fastbridge
http://sr.stateofseo.com/seo-vesti/google-implementacija-ssl-protokola-not-provided-problem/
http://sr.stateofseo.com/napredni-seo/najnovije-promene-google-panda-algoritma/
They are both translated articles (written originally by me on the same website). I've noticed those +1's (476 on both articles) when my website received a penalty for "SEO" keyword on Google.rs (Serbian Google) and I'm now on the 11th page.
Other keywords still rank just fine. Not cool, right? Now, I think there could be two solutions:
First one is to remove my inner link that's pointing to my homepage with "SEO" anchor, and hope for the best. Second one is to completely remove/delete those two articles and wait for Google to reindex the website and hopefully remove my ban. Do you guy have some other ideas how can I fix this or remove / disavow those +1 or somehow explain to the Google crew / algo that I'm just a humble SEO without any evil thoughts? 🙂 Thank you in advance.0 -
I need de-spam help/advice
For one of my sites I am working on I outsourced SEO about 3 years ago. One of the "tricks" the SEO used at the time was to pay for several Blog posts to be "sponsored" by this web site using exact match keywords for the domain. 1 Where do I look to determine the spammy links pointing to this site? 2 Have you had success getting rid of these bad links?
White Hat / Black Hat SEO | | kadesmith0 -
Google-backed sites' link profiles
Curious what you SEO people think of the link profiles of these (high-ranking) Google-backed UK sites: http://www.opensiteexplorer.org/domains?site=www.startupdonut.co.uk http://www.opensiteexplorer.org/domains?site=www.lawdonut.co.uk http://www.opensiteexplorer.org/domains?site=www.marketingdonut.co.uk http://www.opensiteexplorer.org/domains?site=www.itdonut.co.uk http://www.opensiteexplorer.org/domains?site=www.taxdonut.co.uk Each site has between 40k and 50k inlinks counted in OSE. However, there are relatively few linking root domains in each case: 273 for marketingdonut 216 for startupdonut 90 for lawdonut 53 for itdonut 16 for taxdonut Is there something wrong with the OSE data here? Does this imply that the average root domain linking to the taxdonut site does so with 2857 links? The sites have no significant social media stats. The sites are heavily inter-linked. Also linked from the operating business, BHP Information Solutions (tagline "Gain access to SMEs"). Is this what Google would think of as a "natural" link profile? Interestingly, they've managed to secure links on quite a few UK local authority resources pages - generally being the only commercial website on those pages.
White Hat / Black Hat SEO | | seqal0 -
I've done some link building on my website... why is google showing this?
Hi guys, it seems Google is going crazy as always, basically sometimes i'm ranked first page sometimes i'm not there, not sure if it's because of my link building and Google is indexing the links. At the moment in IE i'm top 3-4 for this keyword however the Title tag is not what I set it to be it's basically taking the product name then adding something after it. (I know google sometimes changes to what they want if they feel its more relevant but it isn't in this case) Not sure if this is normal for my keyword to keep appearing then dissapearing in Google. I noticed in FF my keyword isn't there but in IE it is. I've logged out of my Google account deleted all history/cookies etc. Even checked on my friends computer. Hope this makes sense and i'm not going crazy!
White Hat / Black Hat SEO | | InkCartridgesFast0 -
How many times should one submit the same article to various websites? 1 time? 10 times? What is okay to do with the most recent Panda update?'
For link-building purposes, seemingly it was okay to post the same article to multiple sites for links in the past. However, after the most recent Panda update our thought is that this may not be a good practice. So the question is, how many times is okay to submit an article for link building purposes. Should you always only submit to one site? Is it okay to do more than once? What is the right way to submit for link-building in Google's eyes? Thanks
White Hat / Black Hat SEO | | Robertnweil10 -
Why Proved Spammers are on 1st Google SERP's Results
This question is related exclusively to few proved spammers who have gained 1st Google search results for specific terms in the Greek market, targeting Greek audience. Why he looks spammer and very suspicious? For instance, the site epipla-sofa.gr, sofa.gr, fasthosting.gr and greekinternetmarketing.com look suspicious regarding their building link activities: 1. suspicious spiky link growth 2. several links from unrelated content (unrelated blog posts forom other markets, paid links, hidden links) 3. excessive amount of suspicious link placements (forum profiles, blog posts, footer and sidebar links) 4. Greek anchor text with the keyword within articles written in foreign languages (total spam) 5. Unnatural anchor text distribution (too many repetitions) So the main question is: Why Google is unable to recognize/trace some of these (or even all) obvious spamming tactics and still these spammy sites as shwon below reside on the 1st Google.gr SERPs. Examples of spam sites according to their link building history: www.greekinternetmarketing.com www.epipla-sofa.gr www.fasthosting.gr www.sofa.gr All their links look very similar. They use probably software to build links, or even hack authority sites and leave hidden links (really dont know how they could do that). Could you please explain or share similar issues? Have you ever found any similar cases in your industry, and how did you tackle it? We would appreciate your immediate attention to this matter. Regards, George
White Hat / Black Hat SEO | | Clickwisegr0