Error Code 612: Error response for robots.txt

[david.weston]

Hi,

We are getting Error Code 612: Error response for robots.txt in our crawl but everything looks to be ok with the robots file.

Can you confirm what is wrong?

Thanks

[KristinaKeyser]

Hi Wendy! Kristina from Moz's Help Team here - I wanted to chime in here as I had a chance to look over your site and it appears that your site is blocking AWS.

We are getting a "403 Forbidden Error" when attempting to access your site as AWS: http://screencast.com/t/P858BVEQk

Additionally, this 3rd party tool, hurl.it which also uses AWS is getting an Internal Server Error when trying to access your site as well: http://screencast.com/t/N5T822Zpdo

Please re-connect with your developers and make sure they're addressing the issue with your site blocking AWS and that should resolve the issue you're seeing currently in Moz.

I hope this helps but please let us know if there's more we can assist with!
-Kristina

[SOM24]

Chiaryn can you take a look at something?  I am getting a 612 error on this website: www.seminolepowersports.com the developer is telling me there is nothing wrong from what they can see and they are saying MOZ is using AWS server and they have blocked the AWS server from the site.  Questions for you:

1. Does MOZ use AWS server? and could it be the site is blocking then?
2. or is the site confusing the Moz bot, Roger?

Thank you for your input.
Wendy

[ChiarynMiranda]

Hey David, thanks for your question.

I took a look at your campaign and it seem that this is a bit different than the case in the previous post that Thomas linked to in his reply.

It actually looks like you have a redirect loop in place which could be confusing our bot, Roger. The robots.txt page redirects to the www version of the homepage, which redirects to an /en/home subfolder, which redirects to /en/home?r=US. You can verify this using the third party tool https://httpstatus.io/ (http://www.screencast.com/t/pk4fvGXJ1).

I can't say with entire certainty that this is causing the error message you are seeing, as I have never seen a redirect loop on the robots.txt file for a site, but I do know that the crawler will only follow two redirects and any more redirects than that will prevent us from accessing the page, which would likely be reported as an error with the robots.txt.

I would recommend fixing that so that you have only one 301 in place that points to a 200 page or by having the robots.txt file for the site respond with a 200 status. This will need to be done by your site administrator or developer.

[ThomasHarvey]

I'm not sure if this is of any help to you? https://moz.com/community/q/without-robots-txt-no-crawling