What's the best way to eliminate "429 : Received HTTP status 429" errors?
-
My company website is built on WordPress. It receives very few crawl errors, but it do regularly receive a few (typically 1-2 per crawl) "429 : Received HTTP status 429" errors through Moz.
Based on my research, my understand is that my server is essentially telling Moz to cool it with the requests. That means it could be doing the same for search engines' bots and even visitors, right? This creates two questions for me, which I would greatly appreciate your help with:
-
Are "429 : Received HTTP status 429" errors harmful for my SEO? I imagine the answer is "yes" because Moz flags them as high priority issues in my crawl report.
-
What can I do to eliminate "429 : Received HTTP status 429" errors?
Any insight you can offer is greatly appreciated!
Thanks,
Ryan -
-
I have a customer that is using GoDaddy website hosting (at least according to BuiltWith) and I'm experiencing this same issue.
Any updates on this experiment from user rsigg? I'd love to know if I can remove this from my customer's robots file...
FWIW, Netrepid is a hosting provider for colocation, infrastructure and applications (website hosting being considered an application) and we would never force a crawl delay on a Wordpress install!
Not hating on the other hosting service providers... #justsayin
-
I am also on the same hosting and they have not been able to help with the 429. I have now started getting 429 errors when I attempt to login. Definitely something wrong with wp premium hosting.
-
Interesting. I look forward to hearing your results, as my robots.txt file is also set to:
Crawl-delay: 1.
-
We host on Media Temple's Premium WordPress hosting (which I do not recommend, but that's another post for another place), and the techs there told me that it could be an issue with the robots.txt file:
"The issue may be with the settings in the robots.txt file. It looks fine to me but the "Crawl-delay" line might be causing issues. I understand. For the most part, crawlers tend to use robots.txt to determine how to crawl your site, so you may want to see if Moz requires some special settings in there to work correctly."
Ours is set to:
Crawl-delay: 1
I haven't tried changing these values yet in our file, but may experiment with this very soon. If I get results, I'll post back here as well as start a new forum thread.
-
Chase,
They ran a bunch of internal diagnostic tools on my site, and were unable to replicate the 429 errors. They ended up telling me exactly what they told you. I haven't noticed any issues with my site's rankings, or any flags in Webmaster Tools, so it looks like they are right so far. I just hate logging into Moz and seeing all those crawl errors!
-
What'd they say Ryan? Having the same issue and just contacted Godaddy who told me that basically Moz's software is pinging my client's server too frequently so Godaddy is temporarily blocking their IP. They said it's not a concern though as they would never block Google from pinging/indexing the site.
-
Many thanks - I will contact them now!
-
Contact your host and ask let them know about the errors. More than likely they have mod_sec enabled to limit request rates. Ask them to up the limit that you are getting 429 errors from crawlers and you do not want them.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz and HubSpot SSL - crawl error?
I'm getting an error message when Moz tries to crawl my site, however when I check in Google Search Console, they return no errors. Our site is hosted on HubSpot. Is Moz still having trouble crawling HubSpot sites that have enabled their SSL? I read an article that this should have been corrected in early 2017, but I'm getting an error.
Moz Pro | | jennygriffin0 -
What's more valuable: new content or optimizing old content
We are a niche legacy print publication that's been around for close to 20 years. Recently, we combined several old sites in one new responsive site. We have over 7,000 articles -many of which are evergreen and can be repurposed when needed. Most of the old pieces although published, have not been optimized for SEO. However, as we create new pieces, we optimize them for search and social and they tend to get more organic traffic. Where we're torn is on how much we should balance our limited editorial resources between cleaning up and optimizing our extensive archive to improve our organic reach, vs. pumping out new original pieces each week. I realize that without a lot of data the answers will be varied - I guess I'm looking for a best practices approach for content publishers. If it helps at all, our main conversion goal is selling subscriptions to our print and digital publications. We know that organic traffic tends to be more engaged than our social referrals. Unfortunately, due to the nature of the magazine fulfilment business, it's tough to know which channels convert better. Thanks!
Moz Pro | | RicardoSalcedo0 -
What is the best free 'contact finder' tool?
What is the best free 'contact finder' tool? By contact finder I mean a tool that can search multiple websites and display the contact details of each site. Why do Moz not provide such a tool? Thanks
Moz Pro | | conor10050 -
Massive Rank Changes in Bing/Yahoo Only - What's up w/ Rank Tracker?
I'll be honest, it's pretty shameful how little attention I give Bing and Yahoo. But something recently caught my eye - I have a client who according to rank tracker is all over the place in yahoo/bing...up 20 positions for some kw's, down 12 for others...meanwhile Google's holding steady, and Google data seems mostly accurate - with one exception of showing 46th position for an exact match branded search that they're #1 for. I know Rank Tracker has been going through some fine tuning, right? I don't manually check Yahoo/Bing - I can't imagine those #'s are accurate though. Am I right to assume it's incorrect data? Just curious, any insight would be much appreciated. Thanks
Moz Pro | | SVmedia0 -
"Loading" error in Open Site Explorer
The "top pages" keeps hanging while trying to retrieve social metrics "FB Shares" "Tweets" and "Google +1". I've tried in Chrome and Firefox. Can I just turn these off? ose-hanging.jpg
Moz Pro | | cpaddock0 -
Is there a way to get the domain authority of a subdomain?
For example, www.storage-mart.com and us.storage-mart.com both show the same DA in OSE. I believe it's the DA of the root domain. How can I compare the DA of the 'www' and 'us' subdomains?
Moz Pro | | Whitespark0 -
Difference in data between http://pro.seomoz.org/tools/keyword-difficulty and http://lsapi.seomoz.com/linkscape/url-metrics/
Hi, Has any once else experienced any difference in data between http://lsapi.seomoz.com/linkscape/url-metrics/ and http://pro.seomoz.org/tools/keyword-difficulty Please look at the attached image. For "http://www.webmd.com/diet/guide/choosing-weight-loss-program" and "http://www.freedieting.com/" page authority and domain authority match exactly. But for "http://www.fitnessmagazine.com/weight-loss/plans/" data does not match. The data from "http://lsapi.seomoz.com/linkscape/url-metrics/" was retrieved brely 60 seconds latter after data from "http://pro.seomoz.org/tools/keyword-difficulty". We used our custom app for retrieve data from "http://lsapi.seomoz.com/linkscape/url-metrics/". The columns were matched against the specs given in "http://apiwiki.seomoz.org/w/page/13991153/URL-Metrics-API". We are retrieving following columns 1)ut(Title) 2)ueid(External Links) 3)uid(Links) 4)umrp(mozRank) 5)upa(Page Authority) 6)pda(Domain Authority) Any help will be greatly appreciated. zvFif.jpg
Moz Pro | | claytons0 -
How to tell where a competitor's Facebook Shares are coming from?
Is there a way to tell where a competitor's facebook shares and likes are coming from? How to also tell what ads they have running in Facebook and their spend? I am looking at OSE's data and its telling me a competitor has 8.5K Facebook shares and 1K facebook likes. I go to their facebook page and it has about 25 likes. This site should not have anywhere close to the facebook shares/likes its receiving so wondering where they are getting their boosted traffic from.
Moz Pro | | rjb6270