What's the best way to eliminate "429 : Received HTTP status 429" errors?
-
My company website is built on WordPress. It receives very few crawl errors, but it do regularly receive a few (typically 1-2 per crawl) "429 : Received HTTP status 429" errors through Moz.
Based on my research, my understand is that my server is essentially telling Moz to cool it with the requests. That means it could be doing the same for search engines' bots and even visitors, right? This creates two questions for me, which I would greatly appreciate your help with:
-
Are "429 : Received HTTP status 429" errors harmful for my SEO? I imagine the answer is "yes" because Moz flags them as high priority issues in my crawl report.
-
What can I do to eliminate "429 : Received HTTP status 429" errors?
Any insight you can offer is greatly appreciated!
Thanks,
Ryan -
-
I have a customer that is using GoDaddy website hosting (at least according to BuiltWith) and I'm experiencing this same issue.
Any updates on this experiment from user rsigg? I'd love to know if I can remove this from my customer's robots file...
FWIW, Netrepid is a hosting provider for colocation, infrastructure and applications (website hosting being considered an application) and we would never force a crawl delay on a Wordpress install!
Not hating on the other hosting service providers... #justsayin
-
I am also on the same hosting and they have not been able to help with the 429. I have now started getting 429 errors when I attempt to login. Definitely something wrong with wp premium hosting.
-
Interesting. I look forward to hearing your results, as my robots.txt file is also set to:
Crawl-delay: 1.
-
We host on Media Temple's Premium WordPress hosting (which I do not recommend, but that's another post for another place), and the techs there told me that it could be an issue with the robots.txt file:
"The issue may be with the settings in the robots.txt file. It looks fine to me but the "Crawl-delay" line might be causing issues. I understand. For the most part, crawlers tend to use robots.txt to determine how to crawl your site, so you may want to see if Moz requires some special settings in there to work correctly."
Ours is set to:
Crawl-delay: 1
I haven't tried changing these values yet in our file, but may experiment with this very soon. If I get results, I'll post back here as well as start a new forum thread.
-
Chase,
They ran a bunch of internal diagnostic tools on my site, and were unable to replicate the 429 errors. They ended up telling me exactly what they told you. I haven't noticed any issues with my site's rankings, or any flags in Webmaster Tools, so it looks like they are right so far. I just hate logging into Moz and seeing all those crawl errors!
-
What'd they say Ryan? Having the same issue and just contacted Godaddy who told me that basically Moz's software is pinging my client's server too frequently so Godaddy is temporarily blocking their IP. They said it's not a concern though as they would never block Google from pinging/indexing the site.
-
Many thanks - I will contact them now!
-
Contact your host and ask let them know about the errors. More than likely they have mod_sec enabled to limit request rates. Ask them to up the limit that you are getting 429 errors from crawlers and you do not want them.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Is there a way for me to find out how a keyword would rank if it were on a specific site?
Is there a way for me to find out how a keyword would rank if it were on a specific site? For example, lets say that XYZ.com does not have the keyword "ABC". Is there a way for me to find out how the keyword "ABC" would rank if it were on XYZ.com?
Moz Pro | | TurboH0 -
What's more valuable: new content or optimizing old content
We are a niche legacy print publication that's been around for close to 20 years. Recently, we combined several old sites in one new responsive site. We have over 7,000 articles -many of which are evergreen and can be repurposed when needed. Most of the old pieces although published, have not been optimized for SEO. However, as we create new pieces, we optimize them for search and social and they tend to get more organic traffic. Where we're torn is on how much we should balance our limited editorial resources between cleaning up and optimizing our extensive archive to improve our organic reach, vs. pumping out new original pieces each week. I realize that without a lot of data the answers will be varied - I guess I'm looking for a best practices approach for content publishers. If it helps at all, our main conversion goal is selling subscriptions to our print and digital publications. We know that organic traffic tends to be more engaged than our social referrals. Unfortunately, due to the nature of the magazine fulfilment business, it's tough to know which channels convert better. Thanks!
Moz Pro | | RicardoSalcedo0 -
Is there a easy way to see what pages are crawled?
Hello! Like the questions says... Is there a easy way to see what pages are crawled? I don't mean the ones that have issues, but just the ones that have been crawled? Regards,
Moz Pro | | MattDG0 -
What is the best practice for replacing an old xml sitemap?
I have an existing xml sitemap that my website developer loaded, however I don't think its set up properly. What is the best practice for replacing an old xml sitemap? Is there anything I should be concerned about?
Moz Pro | | webestate0 -
What does the "Internal Links" data in the Keyword Difficulty SERP report represent?
What does the "Internal Links" data in the Keyword Difficulty SERP report represent? Thank you! QocBS
Moz Pro | | richpalpine0 -
How many of my linked pages should I redirect (301's)
I'm moving my store to a new site and will have a much friendlier but completely different URL structure. I used Open Site Explorer to find inbound links to 513 pages and have done about half so far. The remaining pages have one link each at a page authority of 27 or less - but there are still 250+ of them. I have to manually view each old page, search for the product on the new site, and enter the redirect as there is no way to translate old URL's to new ones. How important is it for rankings to redirect the remaining 250 or so pages?
Moz Pro | | agirlandamac0 -
Why am I getting duplicate content errors on same page?
In the SEOmoz tools I am getting multiple errors for duplicate page content and duplicate page titles for one section on my site. When I check to see which page has the duplicate title/content the url listed is exactly the same. All sections are set up the same, so any ideas on why I would be getting duplication errors in just this one section and why they would say the errors are on the same page (when I only have one copy uploaded on the server)?
Moz Pro | | CIEEwebTeam0 -
Is there a "manager view" where I can share just one campaign?
I have an account where we manager several different domains, but I need a manager to only see ONE of those campaigns, not all of them. Is there a manager view where I an share just that one campaign?
Moz Pro | | rivershark0