What's the best way to eliminate "429 : Received HTTP status 429" errors?
-
My company website is built on WordPress. It receives very few crawl errors, but it do regularly receive a few (typically 1-2 per crawl) "429 : Received HTTP status 429" errors through Moz.
Based on my research, my understand is that my server is essentially telling Moz to cool it with the requests. That means it could be doing the same for search engines' bots and even visitors, right? This creates two questions for me, which I would greatly appreciate your help with:
-
Are "429 : Received HTTP status 429" errors harmful for my SEO? I imagine the answer is "yes" because Moz flags them as high priority issues in my crawl report.
-
What can I do to eliminate "429 : Received HTTP status 429" errors?
Any insight you can offer is greatly appreciated!
Thanks,
Ryan -
-
I have a customer that is using GoDaddy website hosting (at least according to BuiltWith) and I'm experiencing this same issue.
Any updates on this experiment from user rsigg? I'd love to know if I can remove this from my customer's robots file...
FWIW, Netrepid is a hosting provider for colocation, infrastructure and applications (website hosting being considered an application) and we would never force a crawl delay on a Wordpress install!
Not hating on the other hosting service providers... #justsayin
-
I am also on the same hosting and they have not been able to help with the 429. I have now started getting 429 errors when I attempt to login. Definitely something wrong with wp premium hosting.
-
Interesting. I look forward to hearing your results, as my robots.txt file is also set to:
Crawl-delay: 1.
-
We host on Media Temple's Premium WordPress hosting (which I do not recommend, but that's another post for another place), and the techs there told me that it could be an issue with the robots.txt file:
"The issue may be with the settings in the robots.txt file. It looks fine to me but the "Crawl-delay" line might be causing issues. I understand. For the most part, crawlers tend to use robots.txt to determine how to crawl your site, so you may want to see if Moz requires some special settings in there to work correctly."
Ours is set to:
Crawl-delay: 1
I haven't tried changing these values yet in our file, but may experiment with this very soon. If I get results, I'll post back here as well as start a new forum thread.
-
Chase,
They ran a bunch of internal diagnostic tools on my site, and were unable to replicate the 429 errors. They ended up telling me exactly what they told you. I haven't noticed any issues with my site's rankings, or any flags in Webmaster Tools, so it looks like they are right so far. I just hate logging into Moz and seeing all those crawl errors!
-
What'd they say Ryan? Having the same issue and just contacted Godaddy who told me that basically Moz's software is pinging my client's server too frequently so Godaddy is temporarily blocking their IP. They said it's not a concern though as they would never block Google from pinging/indexing the site.
-
Many thanks - I will contact them now!
-
Contact your host and ask let them know about the errors. More than likely they have mod_sec enabled to limit request rates. Ask them to up the limit that you are getting 429 errors from crawlers and you do not want them.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Found hidden pages of outbound links created via ex-SEO consultant. Best way to detect other possible problems?
We paid for an SEO contract in addition to our new website design (same company did both) and after 12 months cancelled the SEO. I have been very suspicious ever since of our bad page rank and general lack of traffic (despite my efforts) and today found a hidden page of outbound links. Currently in shock that this happened although my own fault for lack of due diligence. The SEO consultants were very unhappy that I cancelled the contract so I am worried about the extent of bad links or negative google juice they may have created (god knows what else they may have done). So my questions are: How can I detect any other (potentially hidden) problems? How can I recover from this - any right/wrong way to approach google? What is the best way to bring this up with the SEO consultants? Thank you in advance.
Moz Pro | | marketing-gal0 -
Why are my "Link Analysis" not updated in all my accounts for the third time this year?
The details on the Link Analysis page are the same as 1,5 week ago for all my 14 accounts.
Moz Pro | | Takeaway0 -
How do you guys/gals define a 'row?'
I have a question about calls to the API and how these are measured. I noticed that the URL Metrics calls allow a batch of multiple URLs. We're in a position where we need link data for multiple websites; can we request a single row of data with link information for multiple URLs, or do we need to request a unique row for each URL?
Moz Pro | | ssimburg0 -
Batch lookup domain authority on list of URL's?
I found this site the describes how to use excel to batch lookup url's using seomoz api. The only problem is the seomoz api times out and returns 1 if I try dragging the formula down the cells which leaves me copying, waiting 5 seconds and copying again. This is basically as slow as manually looking up each url. Does anyone know a workaround?
Moz Pro | | SirSud1 -
What Exactly Does "Linking Root Domains" mean??
What Exactly Does "Linking Root Domains" mean?? And how does it affect your ranking for certain Keywords?? Thanks
Moz Pro | | Caseman54 -
"Wed" flag in the ranking repport
Hi seomoz, Sorry to bother you with that "noobie" question but i was wondering what can the wed flag" mean in the repport ? A crawl just ran this morning so i guess it's not some kind of calendar "issue" unless the crawl don't run on every keywords ?? Thanks for the enlightenment 😉
Moz Pro | | mywebshop0 -
OSE Advanced Reports best practices
I'm curious how people use the OpenSiteExplorer Advanced Reports tab. It seems very powerful. What do you use it for? In particular, I see that it has choices for 'same C block' and 'different C block'. Those seem useful to find C blocks that my competitors have links from that I do not, but I'm not totally clear on how to construct the query. Any help or best practices would be appreciated. Thanks!
Moz Pro | | scanlin0 -
What is the best/your favorite management software?
Currently my company uses basecamp with a splash of google docs a little bit of Paymo, and Raventools for project management. (We use other tools like SEOmoz, but not so much for management as SEO tasks) The ideal features we are looking for are: Collaborative document editing and sharing(Google Docs) Task lists and project organization (basecamphq) Time Tracking on a per task basis (Paymo) (SEO tracking software is great too, but it doesn't have to be integrated with the project management directly. Currently using Raventools and SEOMoz along with some smaller tools.) The project management side can be completely separate from the SEO tool side, but it would be great if there was one simple interface that all of this could be done from. Any suggestions? Are there features I'm missing in my current software that could bring them up to this level?
Moz Pro | | MarloSchneider0