Error Code 612: Error response for robots.txt
-
Hi,
We are getting Error Code 612: Error response for robots.txt in our crawl but everything looks to be ok with the robots file.
Can you confirm what is wrong?
Thanks
-
Hi Wendy! Kristina from Moz's Help Team here - I wanted to chime in here as I had a chance to look over your site and it appears that your site is blocking AWS.
We are getting a "403 Forbidden Error" when attempting to access your site as AWS: http://screencast.com/t/P858BVEQk
Additionally, this 3rd party tool, hurl.it which also uses AWS is getting an Internal Server Error when trying to access your site as well: http://screencast.com/t/N5T822Zpdo
Please re-connect with your developers and make sure they're addressing the issue with your site blocking AWS and that should resolve the issue you're seeing currently in Moz.
I hope this helps but please let us know if there's more we can assist with!
-Kristina -
Chiaryn can you take a look at something? I am getting a 612 error on this website: www.seminolepowersports.com the developer is telling me there is nothing wrong from what they can see and they are saying MOZ is using AWS server and they have blocked the AWS server from the site. Questions for you:
- Does MOZ use AWS server? and could it be the site is blocking then?
- or is the site confusing the Moz bot, Roger?
Thank you for your input.
Wendy -
Hey David, thanks for your question.
I took a look at your campaign and it seem that this is a bit different than the case in the previous post that Thomas linked to in his reply.
It actually looks like you have a redirect loop in place which could be confusing our bot, Roger. The robots.txt page redirects to the www version of the homepage, which redirects to an /en/home subfolder, which redirects to /en/home?r=US. You can verify this using the third party tool https://httpstatus.io/ (http://www.screencast.com/t/pk4fvGXJ1).
I can't say with entire certainty that this is causing the error message you are seeing, as I have never seen a redirect loop on the robots.txt file for a site, but I do know that the crawler will only follow two redirects and any more redirects than that will prevent us from accessing the page, which would likely be reported as an error with the robots.txt.
I would recommend fixing that so that you have only one 301 in place that points to a 200 page or by having the robots.txt file for the site respond with a 200 status. This will need to be done by your site administrator or developer.
-
I'm not sure if this is of any help to you? https://moz.com/community/q/without-robots-txt-no-crawling
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved What would the exact text be for robots.txt to stop Moz crawling a subdomain?
I need Moz to stop crawling a subdomain of my site, and am just checking what the exact text should be in the file to do this. I assume it would be: User-agent: Moz
Getting Started | | Simon-Plan
Disallow: / But just checking so I can tell the agency who will apply it, to avoid paying for their time with the incorrect text! Many thanks.0 -
Standard Syntax in robots.txt doesn't prevent Moz bot from crawling
A client is getting many false positive site crawl errors for things like duplicate titles and duplicate content on pages that include /tag/ in the URL. An example is https://needquest.com/place_tag/autism-spectrum-disorder/page/4/ To resolve this we have set up a disallow statement in the robots.txt file that says
Getting Started | | btreloar
Disallow: /page/ For some reason this appears not to work, as the site crawl errors continue to list pages like this. Does anyone understand why that would be and what we need to do to properly disallow crawling these pages?0 -
SSL - green padlock but Moz say there's an 804 error?
Hi, my site has a green padlock and no SSL errors but Moz are reporting an 804 error. I use CloudFlare with fairly complex settings. I've read this thread but it's quite old and I don't understand which parts of it are still valid. I'd love to know whether this can be sorted before I spend hours setting up Moz's features as if they can't crawl my site then I would obviously need to cancel my subscription. Thanks
Getting Started | | Barn2Plugins0 -
803 Error
I started receiving an 803 error last week for two campaigns and our web dev team has not done any updates to trigger this. Any help on getting this fixed would be greatly appreciated! Paul
Getting Started | | paul_pm0 -
0 status codes
Hi, I am hoping someone could help explain - I have a screaming frog account and ran a crawl using it, it returned a lot of 0 status codes 'connection timeout', 'DNS lookup failed' and 'Connection Refused'. However, the SEO MOZ Crawl doesn't detect any of this. I am fairly new to these tools, do you know why this is? Thanks a lot! 🙂
Getting Started | | eLab_London0 -
After fixing Crawl Errors, how long does it take to for Moz or Google to re-crawl a website?
Last night I found out through Moz that my robots.txt file was blocking any crawling of my website. I fixed the issue. Now do I just sit and wait?
Getting Started | | cmc-interactive0 -
Unable to connect to Google Analytics-Bad Request Error 400
Trying to create a campaign but the step 3 of connection to Google Analytics is giving error. Please suggest. Thanks
Getting Started | | Vaneet.Gupta0 -
Need to Add Moz code?
Do i need to add tracking code in my website, for Moz tool to analyze my website. Please someone let me know.
Getting Started | | TOBOC0