What's the best way to eliminate "429 : Received HTTP status 429" errors?
-
My company website is built on WordPress. It receives very few crawl errors, but it do regularly receive a few (typically 1-2 per crawl) "429 : Received HTTP status 429" errors through Moz.
Based on my research, my understand is that my server is essentially telling Moz to cool it with the requests. That means it could be doing the same for search engines' bots and even visitors, right? This creates two questions for me, which I would greatly appreciate your help with:
-
Are "429 : Received HTTP status 429" errors harmful for my SEO? I imagine the answer is "yes" because Moz flags them as high priority issues in my crawl report.
-
What can I do to eliminate "429 : Received HTTP status 429" errors?
Any insight you can offer is greatly appreciated!
Thanks,
Ryan -
-
I have a customer that is using GoDaddy website hosting (at least according to BuiltWith) and I'm experiencing this same issue.
Any updates on this experiment from user rsigg? I'd love to know if I can remove this from my customer's robots file...
FWIW, Netrepid is a hosting provider for colocation, infrastructure and applications (website hosting being considered an application) and we would never force a crawl delay on a Wordpress install!
Not hating on the other hosting service providers... #justsayin
-
I am also on the same hosting and they have not been able to help with the 429. I have now started getting 429 errors when I attempt to login. Definitely something wrong with wp premium hosting.
-
Interesting. I look forward to hearing your results, as my robots.txt file is also set to:
Crawl-delay: 1.
-
We host on Media Temple's Premium WordPress hosting (which I do not recommend, but that's another post for another place), and the techs there told me that it could be an issue with the robots.txt file:
"The issue may be with the settings in the robots.txt file. It looks fine to me but the "Crawl-delay" line might be causing issues. I understand. For the most part, crawlers tend to use robots.txt to determine how to crawl your site, so you may want to see if Moz requires some special settings in there to work correctly."
Ours is set to:
Crawl-delay: 1
I haven't tried changing these values yet in our file, but may experiment with this very soon. If I get results, I'll post back here as well as start a new forum thread.
-
Chase,
They ran a bunch of internal diagnostic tools on my site, and were unable to replicate the 429 errors. They ended up telling me exactly what they told you. I haven't noticed any issues with my site's rankings, or any flags in Webmaster Tools, so it looks like they are right so far. I just hate logging into Moz and seeing all those crawl errors!
-
What'd they say Ryan? Having the same issue and just contacted Godaddy who told me that basically Moz's software is pinging my client's server too frequently so Godaddy is temporarily blocking their IP. They said it's not a concern though as they would never block Google from pinging/indexing the site.
-
Many thanks - I will contact them now!
-
Contact your host and ask let them know about the errors. More than likely they have mod_sec enabled to limit request rates. Ask them to up the limit that you are getting 429 errors from crawlers and you do not want them.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz shows duplicate content, but URL's are tagged with campaign tags
Crawl diagnostics shows a lot of pages with duplicate content, but when I check the details, I see that it lists the same page but the url contains a campaign tag, so it's not really another page that is serving identical content... Is there a way to remove these pages out of the Crawl Diagnostics?
Moz Pro | | jorisbrabants0 -
What is the best way to track our rank vs competitors?
I have 181 keywords that I have been asked to track rank on. So first I will track against our url then our competitors. With a 250 word limit will that count for every url I want to track? Is there any other way to do this than using rank tracker?
Moz Pro | | Sika220 -
Why isn't me SellingRestaurants campaign operating???
I resign a week ago and I'm still waiting for my SellingRestaurants.com campaign to run???
Moz Pro | | restaurantbroker0 -
Best Way to Seemlessly Use Moz and Google Analytics
Hi All, I have Moz Pro and I am beginning to explore Moz Analytics. I currently use Google Analytics for pretty much everything and I am almost at expert level at using the GA interface so I'm ready to learn a new tool and have a second source of data! My question is: what's the best way to get the most of Moz Analytics and GA congruently? Off hand I know that Moz's keyword, rank, and link data, including competitive insights, is more comprehensive than GA. But how else can I use Moz and get the most out of the Moz Analytics? Looking to really dive in and discuss how we can make use of both! Thanks - Vee
Moz Pro | | Symmetri0 -
Crawl Errors and Notices drop to zero
Hi all, After setting up a campaign in Moz the crawl is successful and it showed the Errors and Warnings in crawl diagnostics (each one had about 40-50), but after a few days the number dropped to zero. Only the "notices" seems to stay normal, with a slight drop since the campaign set up, but not dropping to zero. I set this campaign up in a colleague's account and the same thing happened shortly after set up. I didn't find any Q&A already posted so any insight is appreciated!
Moz Pro | | Vanessa120 -
What's the best way for SEO newbie to analyze & fix a site after being hit by Panda?
Hi, I have a prospective client who was in the top 3 on Google for two of their primary keywords. They fell way back in the rankings immediately after Panda was rolled out on September 27, 2012. Two weeks ago, they were at position #118 for one keyword. After looking for them in Rank Checker today, they cannot be found in Google at all. Here's my question. Because of the "bad links" (some pointing to Porno sites)... what's the possibility that this situation cannot be fixed? I don't know... maybe I'm asking an irrelevant question. I'm attempting to assess the situation so I can go back and present my findings to the prospective client. I'm committed to understanding what's going on with their website so I can assess the situation properly. Fixing their problem starts with a correct assessment. They have a ranking problem, and I know I can fix that... IF all their site needs is white hat <acronym title="Search Engine Optimization">SEO</acronym>. What I DON'T want is... to sell them <acronym title="Search Engine Optimization">SEO</acronym> services, only to find out in 3-6 months... I made an incorrect diagnosis of the problem, and therefore sold them the wrong solution. I know I can close the sale if I can show them with reasonable substantiation that the damage is not beyond repair. I'm familiar with the basics of <acronym title="Search Engine Optimization">SEO</acronym>, but I'm unfamiliar with how "bad linking" might effect the long-term commitment to optimizing. They're wondering if they should start over on another website. I was attempting to do an assessment to better understand if my typical approach on this site would be sufficient. Also... I wanted to get an assessment/report to show them something to substantiate my conclusion(s) about their website. If Open Site Explorer is sufficient to do the link analysis... great. At least I know I'm working with the right tool. All I have to do is learn how to use the tool quickly. At this point... I'm not sure which tool would be helpful. So... can you speak to the following 2 questions: 1.) How do you know when ranking problem is beyond fixing? 2.) What software/tool is ideal for doing some link analysis in order to assess the problem, and prescribe a solution? Thanks so much! Ramon
Moz Pro | | 48dayscoach0 -
I have a Rel Canonical "notice" in my Crawl Diagnostics report. I'm presuming that means that the spider has detected a rel canonical tag and it is working as opposed to warning about an issue, is this correct?
I know this seems like a really dumb question but the site I'm working on is a BigCommerce one and I've been concerned about canonicalisation issues prior to receiving this report (I'm a SEOmoz pro newbie also!) and I just want to be clear I am reading this notice correctly. I presume this means that the site crawl has detected the rel canonical tag on these pages and it is working correctly. Is this correct?? Any input is much appreciated. Thanks
Moz Pro | | seanpearse0 -
Best Keyword Ranking Report Tool
What's the best keyword ranking report tool that anyone is using currently? I have used CuteRank (booo), Rank Tracker, SEOBook.com Rank Tracker, etc. Cost isn't an issue - I'm looking for a free or a paid tool. I need to be able to export resutls in PDF and Excel format. Thanks for the feedback!
Moz Pro | | ezclickmedia0