Controlling crawl speed/delay through dynamic server-code and 503's
-
Lately i'm experiencing performance trouble caused by bot traffic. Although Googlebot is not the worst (it's mainly bingbot and ahrefsbot), they cause heavy server load from time to time. We run a lot of sites on one server, so heavy traffic on one site impacts other site's performance.
Problem is that 1) I want a centrally managed solution for all sites (per site administration takes too much time), which 2) takes into account total server-load in stead of only 1 site's traffic and 3) controls overall bot-traffic in stead of controlling traffic for one bot. IMO user-traffic should always be prioritized higher than bot-traffic.
I tried "Crawl-delay:" in robots.txt, but Googlebot doesn't support that. Although my custom CMS system has a solution to centrally manage Robots.txt for all sites at once, it is read by bots per site and per bot, so it doesn't solve 2) and 3).
I also tried controlling crawl-speed through Google Webmaster Tools, which works, but again it only controls Googlebot (and not other bots) and is administered per site. No solution to all three of my problems.
Now i came up with a custom-coded solution to dynamically serve 503 http status codes to a certain portion of the bot traffic. What traffic-portion for which bots can be dynamically (runtime) calculated from total server load at that certain moment. So if a bot makes too much requests within a certain period (or whatever other coded rule i'll invent), some requests will be answered with a 503 while others will get content and a 200.
Remaining question is: Will dynamically serving 503's have a negative impact on SEO? OK, it will delay indexing speed/latency, but slow server-response-times do in fact have a negative impact on the ranking, which is even worse than indexing-latency.
I'm curious about your expert's opinions...
-
Hi INU,
I always like avoid using things like 503s as a general rule. There is almost certainly a better way to do it.
What about just using Google webmaster tools and Bing webmaster tools? Regarding HREFs it depends how much you rely on that tool. If you don't use it, then I'd more more likely to just block that bot in robots.txt and make sure Google and Bing are controlled using the appropriate tools in the respective webmaster tools.
To answer your specific point about whether or not 503 can hurt rankings. In general no as long as they are only short-term. A 503 like 404s or any other response code is a natural part of the web, however, Google has said in the past that repetitive 503s can be treated as permanent rather than temporary and in some cases can result in the pages being removed from the index.
I hope this helps,
Craig
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Crawl anamoly issue on Search Console
Has anyone checked the crwal anamoly issue under the index section on Search console? We recently move to a new site and I'm seeing a huge list of excluded urls which are classified as crawl anamoly (they all lead to 404 page). Does anyone know that if we need to 301 redirect all the links? Is there any other smarter/ more efficiently way to deal with them like set up canonical link (I thought that's what they're used for isn't it?) Thanks!
White Hat / Black Hat SEO | | greenshinenewenergy0 -
Is a Link Wheel Safe If I Control the Wheel?
Hi, folks. Our company operates over 50 disease-specific, nice websites. Currently, we're building resource/landing pages for some therapies and other related topics. One experimental therapy is being investigated across four different disease types: cystic fibrosis, Muscular Dystrophy, Hemophilia, and cancers. We have sites for all of them, and have created original landing pages for each site. Question: is it safe / does it make sense to "link wheel" these pages, especially since the wheel is composed of all our own sites? The other option of course is to simply interlink all of them, but will I get more visibility with a cyclical linking scheme? I'd love to hear your thoughts on this. Thanks!
White Hat / Black Hat SEO | | Michael_Nace1 -
Subtle line of asking links for money/service/benefits
Hello here, I am putting down a link building strategy according to the latest "good practices" and Google recommendations, but I find myself often confused. For example, I'd like to implement the technique suggested by Rand on his article below: https://moz.com/blog/headsmacking-tip-1-link-requests-in-order-confirmation-emails But if you look at the comments, a user suggests to "ask for links in exchange of discounts", and everyone there applaud him for the idea (Rand included). But, wait a second... am I the only one realizing that now days Google discourage to ask for links for "money, services, or any other kind of 'offered' benefit"? So.. where to draw the line here? Here are other examples that I am not sure are "safe" in link building: 1. Ask for links in exchange of a free Membership on a site (where usually a Membership is sold for a price) 2. Ask for links in exchange of exposure (isn't this a sort of "link exchange"?) 3. Ask for link in exchange of "anything else you can think of", even if necessarily doesn't involve money (i.e. for a "certified site badge", for a free e-book, or anything else) I'd really like to know your thoughts on this very sensitive issue. Thank you in advance to anyone for helping me to understand.
White Hat / Black Hat SEO | | fablau1 -
Bad for SEO to have two very similar websites on the same server?
Is it bad for SEO to have two very similar sites on the same server? What's the best way to set this up?
White Hat / Black Hat SEO | | WebServiceConsulting.com0 -
Looking for recent bad SEO / black hat example such as JC Penney example from 2011
I am giving a presentation in a few weeks and looking for a "what not to do" larger brand example that made poor SEO choices to try and game Google with black hat tactics. Any examples you can point me to?
White Hat / Black Hat SEO | | jfeitlinger0 -
IS http://ezinearticles.com/ good or bad for backlinks?
Hi Everyone, Is http://ezinearticles.com/ any good to use? Thanks
White Hat / Black Hat SEO | | vanplus0 -
What's the best way to set up 301's from an old off-site subdomain to a new off-site subdomain?
We are moving our Online store to a new service and we need to create 301's for all of the old product URLs. Being that the old store was hosted off-site, what is the best way to handle the 301 re-directs? Thanks!
White Hat / Black Hat SEO | | VermilionDesignInteractive0 -
'Stealing' link juice from 404's
As you all know, it's valuable but hard to get competitors to link to your website. I'm wondering if the following could work: Sometimes I spot that a competitor is linking to a certain external page, but he made a typo in the URL (e.g. the competitor meant to link to awesomeglobes.com/info-page/ but the link says aewsomeglobes.com/info-page/). Could I then register the typo domain and 301 it to my own domain (i.e. aewsomeglobes.com/info-page/ to mydomain.com/info-page/) and collect the link juice? Does it also work if the link is a root domain?
White Hat / Black Hat SEO | | RBenedict0