Error Code 612: Error response for robots.txt
-
Hi,
We are getting Error Code 612: Error response for robots.txt in our crawl but everything looks to be ok with the robots file.
Can you confirm what is wrong?
Thanks
-
Hi Wendy! Kristina from Moz's Help Team here - I wanted to chime in here as I had a chance to look over your site and it appears that your site is blocking AWS.
We are getting a "403 Forbidden Error" when attempting to access your site as AWS: http://screencast.com/t/P858BVEQk
Additionally, this 3rd party tool, hurl.it which also uses AWS is getting an Internal Server Error when trying to access your site as well: http://screencast.com/t/N5T822Zpdo
Please re-connect with your developers and make sure they're addressing the issue with your site blocking AWS and that should resolve the issue you're seeing currently in Moz.
I hope this helps but please let us know if there's more we can assist with!
-Kristina -
Chiaryn can you take a look at something? I am getting a 612 error on this website: www.seminolepowersports.com the developer is telling me there is nothing wrong from what they can see and they are saying MOZ is using AWS server and they have blocked the AWS server from the site. Questions for you:
- Does MOZ use AWS server? and could it be the site is blocking then?
- or is the site confusing the Moz bot, Roger?
Thank you for your input.
Wendy -
Hey David, thanks for your question.
I took a look at your campaign and it seem that this is a bit different than the case in the previous post that Thomas linked to in his reply.
It actually looks like you have a redirect loop in place which could be confusing our bot, Roger. The robots.txt page redirects to the www version of the homepage, which redirects to an /en/home subfolder, which redirects to /en/home?r=US. You can verify this using the third party tool https://httpstatus.io/ (http://www.screencast.com/t/pk4fvGXJ1).
I can't say with entire certainty that this is causing the error message you are seeing, as I have never seen a redirect loop on the robots.txt file for a site, but I do know that the crawler will only follow two redirects and any more redirects than that will prevent us from accessing the page, which would likely be reported as an error with the robots.txt.
I would recommend fixing that so that you have only one 301 in place that points to a 200 page or by having the robots.txt file for the site respond with a 200 status. This will need to be done by your site administrator or developer.
-
I'm not sure if this is of any help to you? https://moz.com/community/q/without-robots-txt-no-crawling
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Crawler was not able to access the robots.txt
I'm trying to setup a campaign for jessicamoraninteriors.com and I keep getting messages that Moz can't crawl the site because it can't access the robots.txt. Not sure why, other crawlers don't seem to have a problem and I can access the robots.txt file from my browser. For some additional info, it's a SquareSpace site and my DNS is handled through Cloudflare. Here's the contents of my robots.txt file: # Squarespace Robots Txt User-agent: GPTBot User-agent: ChatGPT-User User-agent: CCBot User-agent: anthropic-ai User-agent: Google-Extended User-agent: FacebookBot User-agent: Claude-Web User-agent: cohere-ai User-agent: PerplexityBot User-agent: Applebot-Extended User-agent: AdsBot-Google User-agent: AdsBot-Google-Mobile User-agent: AdsBot-Google-Mobile-Apps User-agent: * Disallow: /config Disallow: /search Disallow: /account$ Disallow: /account/ Disallow: /commerce/digital-download/ Disallow: /api/ Allow: /api/ui-extensions/ Disallow: /static/ Disallow:/*?author=* Disallow:/*&author=* Disallow:/*?tag=* Disallow:/*&tag=* Disallow:/*?month=* Disallow:/*&month=* Disallow:/*?view=* Disallow:/*&view=* Disallow:/*?format=json Disallow:/*&format=json Disallow:/*?format=page-context Disallow:/*&format=page-context Disallow:/*?format=main-content Disallow:/*&format=main-content Disallow:/*?format=json-pretty Disallow:/*&format=json-pretty Disallow:/*?format=ical Disallow:/*&format=ical Disallow:/*?reversePaginate=* Disallow:/*&reversePaginate=* Any ideas?
Getting Started | | andrewrench0 -
Our crawler was not able to access the robots.txt file on your site
I've submitted my website to be crawled by Moz and done everything I can according to the troubleshooting guides. Please help! https://digitalbutter.co.za/robots.txt
Getting Started | | DigitalButter0 -
Error code 804
URL Page Authority Linking Root Domains Status Code | Error Code 804: HTTPS (SSL) Error Encountered https://www.alleninv.com |
Getting Started | | alleninv0 -
4XX Client Error Solution?
Hello. Could you tell em how I can solve the 4XX client error from the moz crawl? I have the broken links in moz crawler like this: test.com/page-1/test.com. Thanks.
Getting Started | | iulianafun0 -
Error Selecting Search Engine During Moz Campaign Set-Up
Hi, I'm setting up a Moz campaign for a new client, and when I get to the third step, "Choose the search engines for which you would like to track rankings," there's nothing to select. No drop down, or anything, just a blank field under the heading. If I try to advance, I get an error pop-up that says I must select a search engine. Has anyone else run into this problem? What should I do? GlWPAsc zAmzqFf
Getting Started | | tommyKPseo0 -
Where do I enter a promotion code?
Hi, I've been referred from Mailchimp and I have a promo code. It said I could redeem it at the check out, but I didn't find it. How can I enter it now? (I am currently on the 30 days free trial) Thanks!
Getting Started | | Stukers0 -
Getting errors -- Pages with Title Missing or Empty
I am getting a high priority warning that I have pages with title missing or empty, however, when I check the page, there is a title tag <title>Long Term Care Planning</title> . Anyone have any idea what could be doing on? I check the CMS (using AEM - adobe experience manager CQ5) and the title field is filled out as well.
Getting Started | | Laura-Genworth0 -
High Number of Crawl Errors for Blog
Hello All, We have been having an issue with very high crawl errors on websites that contain blogs. Here is a screenshot of one of the sites we are dealing with: http://cl.ly/image/0i2Q2O100p2v . Looking through the links that are turning up in the crawl errors, the majority of them (roughly 90%) are auto-generated by the blog's system. This includes category/tag links, archived links, etc. A few examples being: http://www.mysite.com/2004/10/ http://www.mysite.com/2004/10/17/ http://www.mysite.com/tagname As far as I know (please correct me if I'm wrong!), search engines will not penalize you for things like this that appear on auto-generated pages. Also, even if search engines did penalize you, I do not believe we can make a unique meta tag for auto-generate pages. Regardless, our client is very concerned seeing these high number of errors in the reports, even though we have explained the situation to him. Would anyone have any suggestions on how to either 1) tell Moz to ignore these types of errors or 2) adjust the website so that these errors now longer appear in the reports? Thanks so much! Rebecca
Getting Started | | Level2Designs0