What's the best way to eliminate "429 : Received HTTP status 429" errors?
-
My company website is built on WordPress. It receives very few crawl errors, but it do regularly receive a few (typically 1-2 per crawl) "429 : Received HTTP status 429" errors through Moz.
Based on my research, my understand is that my server is essentially telling Moz to cool it with the requests. That means it could be doing the same for search engines' bots and even visitors, right? This creates two questions for me, which I would greatly appreciate your help with:
-
Are "429 : Received HTTP status 429" errors harmful for my SEO? I imagine the answer is "yes" because Moz flags them as high priority issues in my crawl report.
-
What can I do to eliminate "429 : Received HTTP status 429" errors?
Any insight you can offer is greatly appreciated!
Thanks,
Ryan -
-
I have a customer that is using GoDaddy website hosting (at least according to BuiltWith) and I'm experiencing this same issue.
Any updates on this experiment from user rsigg? I'd love to know if I can remove this from my customer's robots file...
FWIW, Netrepid is a hosting provider for colocation, infrastructure and applications (website hosting being considered an application) and we would never force a crawl delay on a Wordpress install!
Not hating on the other hosting service providers... #justsayin
-
I am also on the same hosting and they have not been able to help with the 429. I have now started getting 429 errors when I attempt to login. Definitely something wrong with wp premium hosting.
-
Interesting. I look forward to hearing your results, as my robots.txt file is also set to:
Crawl-delay: 1.
-
We host on Media Temple's Premium WordPress hosting (which I do not recommend, but that's another post for another place), and the techs there told me that it could be an issue with the robots.txt file:
"The issue may be with the settings in the robots.txt file. It looks fine to me but the "Crawl-delay" line might be causing issues. I understand. For the most part, crawlers tend to use robots.txt to determine how to crawl your site, so you may want to see if Moz requires some special settings in there to work correctly."
Ours is set to:
Crawl-delay: 1
I haven't tried changing these values yet in our file, but may experiment with this very soon. If I get results, I'll post back here as well as start a new forum thread.
-
Chase,
They ran a bunch of internal diagnostic tools on my site, and were unable to replicate the 429 errors. They ended up telling me exactly what they told you. I haven't noticed any issues with my site's rankings, or any flags in Webmaster Tools, so it looks like they are right so far. I just hate logging into Moz and seeing all those crawl errors!
-
What'd they say Ryan? Having the same issue and just contacted Godaddy who told me that basically Moz's software is pinging my client's server too frequently so Godaddy is temporarily blocking their IP. They said it's not a concern though as they would never block Google from pinging/indexing the site.
-
Many thanks - I will contact them now!
-
Contact your host and ask let them know about the errors. More than likely they have mod_sec enabled to limit request rates. Ask them to up the limit that you are getting 429 errors from crawlers and you do not want them.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Sitemap Best Practices
My question is regarding the URL structure best practices of a sitemap. My website allows search any number of ways, i.e. 1. http://www.website.com/category/subcategory/product 2. http://www.website.com/subcategory/product 3. http://www.website.com/product However, I am not sure which structure to use in the sitemap (which is being written manually). I know that for SEO purposes the 3rd option is best as the link is more relevant to that individual product, but the Moz tool states that the home page should have less than 100 links (although Google doesn't penalise for having more) and by writing my entire site in the 3rd way it would result in a lot more links adjoining to the home page. It is either the 2nd or 3rd option, I think, as the 1st category is not keyword specific (rather a generic term, i.e. novelties). Does anyone have experience with this?
Moz Pro | | moon-boots0 -
Error in Moz duplicate content reports
Hi - I've run the Moz campaign on a client's site. Moz is saying that there are duplicate content errors, and when I look at the errors it is showing that they are all to do with the non-www URLs having being duplicated in the www form of the URLs. However this is not the case - all the non-www URLs are all 301 redirected to the www URLs. Is this an error in the Moz tool? Has anybody experienced something similar?
Moz Pro | | rorynatkiel0 -
Branded Keywords: Should I Monitor My Products' Names?
I am setting up my keywords' tags and brand rules. Is there any reason not to consider my products' names as "branded keywords". For instance, if my brand is HealthNut and I have products (that people talk about by name), such as GreenGiant Juice, should I create a brand rule for "GreenGiant Juice"? Granted, most of the time, customers will be searching for "HealthNut GreenGiant Juice", but there must be at least a minority who would search directly for the product itself without using the brand name. Any thoughts are greatly appreciated. Thank you in advance!
Moz Pro | | EliteErikSEO0 -
Metric "Total Links" can somebody explain this metric to me?
Dear colleagues, Who can explain the following to me? Subdomain metrics - Total links The total links is huge compared to the sum of internal and external links. I do not understand this metric. Can somebody help me to explain this the metrich "total links" I have to present these metrics to my customer and do not want to have "don't know" as an answer 😉 Thanks, Alain Nijholt BMC Internet Marketing
Moz Pro | | bmcinternetmarketing0 -
404 errors, but not showing in Google analytics
In my SEOmoz errors there are over a dozen 404 errors listed. However, they are not showing up in Google analytics. How can I find the referring url for the ones showing up here?
Moz Pro | | sakeith0 -
Curious, is there any way when browsing through your keywords to view each individual keyword by the search traffic that term receives?
When looking thru our keywords on our reports in SEOmoz, it would be really helpful to be able to rank keywords by major head terms and long tail terms. Is there any way to have the keyword's search traffic show up in the rankings report?
Moz Pro | | ACLens0 -
In OSE "Followed Linking Root Domains" = "links from homepages"
In OSE's, "Followed Linking Root Domains" are defined as "The number of root domains that have at least one followed link to a page or domain." Does this mean that if one of my competitors has, let's say, 1000 followed linking root domains, they have a link pointing to them on the homepage of 1000 other sites? Thanks for your help!
Moz Pro | | gerardoH0 -
Best way to defeat negative press?
Hi there, One big reason my non profit hired me to tackle SEO for our site is that there is currently a lot of negative stuff out there right now about us. There was a lot of stuff that went down in the past, and currently Googling different names of our org's members, or typing in variations of our name, brings up several sites that are negative and inflammatory. They have been up since the early 2000's, so they have that advantage over us. It's interesting because I'm sort of in a position where I'm doing SEO for the sake of doing SEO, which is sort of cool because I get to learn a lot. We don't really have conversion KPIs or anything, but I'm sure that will come in time. I suppose my question is, what is the best way to tackle this? I mean I know that overall doing great SEO will help, but I'm wondering if there's a specific practice I should be focusing on to defeat a ton of negative stuff that has age over us. I'm also concerned that even if we get our site listed #1 for various searches, there will still be the negative stuff on page 1, just lower ranked. Any thoughts? Thanks for your time!
Moz Pro | | MikeQ-BACC0