What's the best way to eliminate "429 : Received HTTP status 429" errors?
-
My company website is built on WordPress. It receives very few crawl errors, but it do regularly receive a few (typically 1-2 per crawl) "429 : Received HTTP status 429" errors through Moz.
Based on my research, my understand is that my server is essentially telling Moz to cool it with the requests. That means it could be doing the same for search engines' bots and even visitors, right? This creates two questions for me, which I would greatly appreciate your help with:
-
Are "429 : Received HTTP status 429" errors harmful for my SEO? I imagine the answer is "yes" because Moz flags them as high priority issues in my crawl report.
-
What can I do to eliminate "429 : Received HTTP status 429" errors?
Any insight you can offer is greatly appreciated!
Thanks,
Ryan -
-
I have a customer that is using GoDaddy website hosting (at least according to BuiltWith) and I'm experiencing this same issue.
Any updates on this experiment from user rsigg? I'd love to know if I can remove this from my customer's robots file...
FWIW, Netrepid is a hosting provider for colocation, infrastructure and applications (website hosting being considered an application) and we would never force a crawl delay on a Wordpress install!
Not hating on the other hosting service providers... #justsayin
-
I am also on the same hosting and they have not been able to help with the 429. I have now started getting 429 errors when I attempt to login. Definitely something wrong with wp premium hosting.
-
Interesting. I look forward to hearing your results, as my robots.txt file is also set to:
Crawl-delay: 1.
-
We host on Media Temple's Premium WordPress hosting (which I do not recommend, but that's another post for another place), and the techs there told me that it could be an issue with the robots.txt file:
"The issue may be with the settings in the robots.txt file. It looks fine to me but the "Crawl-delay" line might be causing issues. I understand. For the most part, crawlers tend to use robots.txt to determine how to crawl your site, so you may want to see if Moz requires some special settings in there to work correctly."
Ours is set to:
Crawl-delay: 1
I haven't tried changing these values yet in our file, but may experiment with this very soon. If I get results, I'll post back here as well as start a new forum thread.
-
Chase,
They ran a bunch of internal diagnostic tools on my site, and were unable to replicate the 429 errors. They ended up telling me exactly what they told you. I haven't noticed any issues with my site's rankings, or any flags in Webmaster Tools, so it looks like they are right so far. I just hate logging into Moz and seeing all those crawl errors!
-
What'd they say Ryan? Having the same issue and just contacted Godaddy who told me that basically Moz's software is pinging my client's server too frequently so Godaddy is temporarily blocking their IP. They said it's not a concern though as they would never block Google from pinging/indexing the site.
-
Many thanks - I will contact them now!
-
Contact your host and ask let them know about the errors. More than likely they have mod_sec enabled to limit request rates. Ask them to up the limit that you are getting 429 errors from crawlers and you do not want them.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Are the metrics in Moz's SERP analysis relevant in my area?
Our site (https://loco2.com/) specialises in train journey planning at ticket booking, so we want to improve our ranking when people search Google for a specific journey they want to make (eg "London to Barcelona by train"). We have a set of about 400 Guide pages (eg London to Barcelona) which we want people to find when they search for those journeys. I've been trying to find out why some of those pages do better than others in search, and what our competitors are doing better. I was hoping that the SERP analysis and Keyword Reports tool could help me with this, so I ran analyses for a set of the keyword phrases we want to rank for, but I was surprised to find that most of the metrics in both the basic and full reports don't seem to relate to how high in the search result a page ranks. The screengrab attached shows the results for "Brussels to Paris train", which are fairly typical - many of the top-ranking pages have fairly low scores for most metrics, and the metrics don't seem to have much effect on how high they are in the search results. ( I assume N/A means the Page Authority is very low, or is it not calculated for some other reason?) I looked into this a bit further by downloading full reports for 11 keyword searches, and comparing the average scores for each metric with the rank, but I still couldn't see any effects of the metrics on the ranks. If anything, pages with higher Page Authority and Domain Authority were further down in search rankings. Does this mean that link metrics aren't very important for the kinds of pages we're looking at? This makes sense to me, because if I'm searching for tickets from London to Barcelona I want to find a very specific results page which is unlikely to have many links, but I keep reading that link metrics are one of the most important factors in SEO. Am I misunderstanding something? wJ9uUcG
Moz Pro | | MargotLoco20 -
I have a new website and would like to use the keywords my competitors are using and was wondering the best way to take a look using Moz Pro?
The website is www.fapwrap.com - it's a new adult wall decals website. [editor's note: may not be safe for some workplaces] The wall decals we have are obviously a bit more niche than the bigger competitors, but I want to get into the first page to see how it helps us. I'm slowly taking over some kws like "xbox wraps" and "ps4 wraps" but those have low search volume and I feel like "wall decals" will help - though it's highly competitive. Suggestions? I just kinda want to plug in my competition and pick from their list of existing kws..... Ha.
Moz Pro | | JohnnyRoq0 -
ERROR: Too Many on-page links!
User wise, my product menu @LEDSupply.com is user friendly, but I'm concerned that it might be seen by crawlers as bad because of the TOO MANY ON-PAGE LINKS error I am getting in my moz crawl report. Is it really counting all the links in every drop-down menu? If so, is there are resource on how to fix it????
Moz Pro | | saultienut0 -
What is the best approach to handling 404 errors?
Hello All - I'm a new here and working on the SEO on my site www.shoottokyo.com. When I am finding 4xx (Client Errors) what is the best way to deal with them? I am finding an error like this for example: http://shoottokyo.com/2010/11/28/technology-and-karma/ This may have been caused when I updated my permalinks from shoottokyo.com/2011/09/postname to shoottokyo.com/postname. I was using the plug in Permalinks moved permanently to fix them. Sometimes I am able to find http://shoottokyo.com/a-very-long-week/www.newscafe.jp and I can tell that I simply have a bad link to News Cafe and I can go to the post and correct it but in the case of the first one I can't find out where the crawler even found the problem. I'm using Wordpress. Is it best to just use a plugin like 'Redirection' to move the rest that have errors where I cannot find the source of the issue? Thanks Dave
Moz Pro | | ShootTokyo0 -
Is there a easy way to see what pages are crawled?
Hello! Like the questions says... Is there a easy way to see what pages are crawled? I don't mean the ones that have issues, but just the ones that have been crawled? Regards,
Moz Pro | | MattDG0 -
Why are the rankings of my keywords different to rankings from other programmes such as 'Access my Lan' and 'Firefly'? Does SEOMoz count 'Google Places and Maps' in their rank count? Im thinking this could explain the difference
Why are the rankings of my keywords different to rankings from other programmes such as 'Access my Lan' and 'Firefly'? Does SEOMoz count 'Google Places and Maps' in their rank count? Im thinking this could explain the difference
Moz Pro | | john_Digino0 -
Duplicate content due to "Email a Friend" and "PhotoGallery"
Crawl Diagnostics gives me 1650 duplicate page content errors. 800+ are for photo gallery, the link upon which my large image is shown. 800+ are for the Email a Friend form page. Presumably SEOMoz's tool is getting there by following the link inside a product page, which as you can see will differ only by the ProductCode of every product.
Moz Pro | | jotham2
www.completemobilehomesupply.com/PhotoGallery.asp?ProductCode=anchor101
www.completemobilehomesupply.com/EmailaFriend.asp?ProductCode=Shutter001 1 - Is it critical to fix?
I am assuming Yes, but if for some reason the answer is No, please share. 2 - Any idea on how to fix?
The site uses Volusion, fyi, so it may be a limiting factor of the platform. Thank you for your time.
Also if this topic has been previously covered, please link and I'll read there instead.0 -
Most of the time getting error.
Hi, i am getting this error most of the time in linkscape since last month. Sorry dude, no inlinks found matching this criteria. Pl guide is this a bug and the sites I am trying to use linkscape for were having lot of pages crawled earlier by SEOMOZ. Thanks, Preet
Moz Pro | | PreetSibia0