Error Code 612: Error response for robots.txt
-
Hi,
We are getting Error Code 612: Error response for robots.txt in our crawl but everything looks to be ok with the robots file.
Can you confirm what is wrong?
Thanks
-
Hi Wendy! Kristina from Moz's Help Team here - I wanted to chime in here as I had a chance to look over your site and it appears that your site is blocking AWS.
We are getting a "403 Forbidden Error" when attempting to access your site as AWS: http://screencast.com/t/P858BVEQk
Additionally, this 3rd party tool, hurl.it which also uses AWS is getting an Internal Server Error when trying to access your site as well: http://screencast.com/t/N5T822Zpdo
Please re-connect with your developers and make sure they're addressing the issue with your site blocking AWS and that should resolve the issue you're seeing currently in Moz.
I hope this helps but please let us know if there's more we can assist with!
-Kristina -
Chiaryn can you take a look at something? I am getting a 612 error on this website: www.seminolepowersports.com the developer is telling me there is nothing wrong from what they can see and they are saying MOZ is using AWS server and they have blocked the AWS server from the site. Questions for you:
- Does MOZ use AWS server? and could it be the site is blocking then?
- or is the site confusing the Moz bot, Roger?
Thank you for your input.
Wendy -
Hey David, thanks for your question.
I took a look at your campaign and it seem that this is a bit different than the case in the previous post that Thomas linked to in his reply.
It actually looks like you have a redirect loop in place which could be confusing our bot, Roger. The robots.txt page redirects to the www version of the homepage, which redirects to an /en/home subfolder, which redirects to /en/home?r=US. You can verify this using the third party tool https://httpstatus.io/ (http://www.screencast.com/t/pk4fvGXJ1).
I can't say with entire certainty that this is causing the error message you are seeing, as I have never seen a redirect loop on the robots.txt file for a site, but I do know that the crawler will only follow two redirects and any more redirects than that will prevent us from accessing the page, which would likely be reported as an error with the robots.txt.
I would recommend fixing that so that you have only one 301 in place that points to a 200 page or by having the robots.txt file for the site respond with a 200 status. This will need to be done by your site administrator or developer.
-
I'm not sure if this is of any help to you? https://moz.com/community/q/without-robots-txt-no-crawling
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved What would the exact text be for robots.txt to stop Moz crawling a subdomain?
I need Moz to stop crawling a subdomain of my site, and am just checking what the exact text should be in the file to do this. I assume it would be: User-agent: Moz
Getting Started | | Simon-Plan
Disallow: / But just checking so I can tell the agency who will apply it, to avoid paying for their time with the incorrect text! Many thanks.0 -
Error code 804
URL Page Authority Linking Root Domains Status Code | Error Code 804: HTTPS (SSL) Error Encountered https://www.alleninv.com |
Getting Started | | alleninv0 -
901 error code showing url back to back in crawl
Hi Everyone, I'm absolutely dumbfounded about this 901 issue (showing pages with our url back to back). Our site is hosted on Big Commerce: https://www.santabarbarachocolate.com When I look for these pages being crawled I don't find them. I've called BC for help and I can't seem to find a solution or where to turn as to how to fix the issue at hand or even if it matters. Please see below what the Moz crawl shows. Could this be related to Yotpo or some app we have running? Or does this even matter and does it have any influence on rank? Do you have recommendations or ideas? Thanks so much. Pages with Crawl Attempt Error as of Mar 3 URL Page Authority Linking Root Domains Status Code | Error Code 901: DNS Errors Prevented Crawler from Resolving Hostname http://www.santabarbarachocolate.comhttp/www.santabarbarachocolate.com/100-percent-pure-cacao-unsweetened-baking-chocolate -- -- 901 Error Code 901: DNS Errors Prevented Crawler from Resolving Hostname http://www.santabarbarachocolate.comhttp/www.santabarbarachocolate.com/buy-wholesale-bulk-chocolate -- -- 901 Error Code 901: DNS Errors Prevented Crawler from Resolving Hostname http://www.santabarbarachocolate.comhttp/www.santabarbarachocolate.com/organic-chocolate-wholesale | -- | -- | 901 |
Getting Started | | santabarbarachocolate0 -
Error Code 804: HTTPS (SSL) Error Encountered
Recently we bought SSL and moved our site to https. (https://www.kitchenstuff.com) where all link are https even after I see an error "Error Code 804: HTTPS (SSL) Error Encountered". Can anyone help me out? What steps can I take to troubleshoot this error?
Getting Started | | torbett0 -
How Do I Scan My New Site & Grade My Work With The Robots Turned Off? For Pre-Inspection before I launch my Site?
I have a new site that has all the bots turned off so google can't index my site until I'm finished it. I've been working on this site for a couple months now optimizing and I was wondering if there was anyway I can run a preliminary scan on the site for my titles, URLs, Headers, Alt Tags and pretty much anything else that will grade my work and tell me if i did anything wrong? Can MOZ do this with the Bots turned off? Thanks
Getting Started | | Inframan0 -
Error help for newbie please
Hi, I signed up after seeing the videos on Udemy and YouTube (white board Friday) So I've started the free trial and am looking forward to getting my site ranking higher. I've crawled my site www.sussexchef.com and its come back with the following errors (please see below. 608 I'm sure this information is very important but I have no idea how to fix the 608, I found a robots.txt in my directory and deleted it as I think that maybe the problem? I crawled the site twice by accident so will have to wait till tomorrow to find out? 404 I found it quite hard to find the broken links at first but once I realized all the information I needed was in the table I think I got them all. did I miss a tutorial or am I just a little out of my depth here? 503 I have no idea how to fix these, I can click the links and it takes my to that page or file. so how can it be the server down? Or is this because they are links to PDF's? should i convert them to jpegs and give them meta data? I'd be grateful for any help anyone has to offer as I'm keen to learn how to promote my site better. Crawl Error Moz encountered an error on one or more pages on your site608 Page not Decodable as Specified Content EncodingInvestigate the cause of this issue on the Help Hub.Discovered: Sep 2 - 8Crawl Diagnostics Crawl Issue Found: 404 Errors 10% of site pages served 404 errors during the last crawlA high percentage of 404 pages can indicate a problem with the internal link structure.Crawl Diagnostics 404 : Received 404 (Not Found) error response for page. http://sussexchef.com/wedding-caterers.aspx4041215http://sussexchef.comN/AView Issue 404 : Received 404 (Not Found) error response for page. http://sussexchef.com/dinner-party-catering.aspx404115http://sussexchef.comN/AView Issue 404 : Received 404 (Not Found) error response for page. http://sussexchef.com/christmas-party-catering.aspx404115http://sussexchef.comN/AView Issue 404 : Received 404 (Not Found) error response for page. http://sussexchef.com/wedding-cakes.aspx404115http://sussexchef.comN/AView Issue 404 : Received 404 (Not Found) error response for page. http://sussexchef.com/outdoor-catering-specialists.aspx404115http://sussexchef.comN/AView Issue 404 : Received 404 (Not Found) error response for page. http://sussexchef.com/hen-party-cupcake-classes.aspx404115http://sussexchef.comN/AView Issue 404 : Received 404 (Not Found) error response for page. http://sussexchef.com/funeral-caterers.aspx404115http://sussexchef.comN/AView Issue 404 : Received 404 (Not Found) error response for page. http://sussexchef.com/drinks-service.aspx404115http://sussexchef.comN/AView Issue 404 : Received 404 (Not Found) error response for page. http://sussexchef.com/private-party-catering.aspx404115http://sussexchef.comN/AView Issue 404 : Received 404 (Not Found) error response for page. http://sussexchef.com/corporate-catering.aspx404115http://sussexchef.comN/AView Issue 404 : Received 404 (Not Found) error response for page. http://sussexchef.com/caterers.aspx40401http://sussexchef.com/wedding-catering/N/AView Issue 404 : Received 404 (Not Found) error response for page. http://sussexchef.com/funeral-caterers-brighton.aspx Crawl Issue Found: 500 Errors More than 5% of site pages served 500 errors during the last crawlExcessive 500 errors impact search engine indexation. Double check that your website is serving pages properly to both users and crawlers. 503 : Received 503 (Service Unavailable) error response for page. http://sussexchef.com/wp-content/uploads/2013/08/Wedding-Packages-2014.pdf141N/A50302N/A 503 : Received 503 (Service Unavailable) error response for page. http://sussexchef.com/wp-content/uploads/2013/08/Vegetarian-BBQ-Menu.pdf141N/A50301N/A 503 : Received 503 (Service Unavailable) error response for page. http://sussexchef.com/wp-content/uploads/2013/08/AllInclusiveMenuPrices1.pdf141N/A50301N/A 503 : Received 503 (Service Unavailable) error response for page. http://sussexchef.com/Finger%20Buffet.pdf141N/A50301N/A 503 : Received 503 (Service Unavailable) error response for page. http://sussexchef.com/Dessert.pdf141N/A50301N/A 503 : Received 503 (Service Unavailable) error response for page. http://sussexchef.com/wp-content/uploads/2013/08/SummerMenu.pdf141N/A50301N/A 503 : Received 503 (Service Unavailable) error response for page. http://sussexchef.com/HotorColdBuffet.pdf141N/A50301N/A 503 : Received 503 (Service Unavailable) error response for page. http://sussexchef.com/Canape%20Menu.pdf141N/A50301N/A 503 : Received 503 (Service Unavailable) error response for page. http://sussexchef.com/wp-content/uploads/2013/08/Susex-Chef-Xmas-Dinner-artwork-file.pdf141N/A50301N/A 503 : Received 503 (Service Unavailable) error response for page. http://sussexchef.com/BBQ.pdf141N/A50301N/A 503 : Received 503 (Service Unavailable) error response for page. http://sussexchef.com/wp-content/uploads/2013/08/Fun-Finger-Buffets.pdf141N/A50301N/A 503 : Received 503 (Service Unavailable) error response for page. http://sussexchef.com/HogRoast.pdf141N/A50301N/A 503 : Received 503 (Service Unavailable) error response for page. http://sussexchef.com/wp-content/uploads/2013/08/Private-Chef-Dinner-Packages.pdf141N/A50301N/A 503 : Received 503 (Service Unavailable) error response for page. http://sussexchef.com/Salad%20Menu.pdf141N/A50301N/A 503 : Received 503 (Service Unavailable) error response for page. http://sussexchef.com/wp-content/uploads/2013/08/Childrens-Menu.pdf141N/A50301N/A 503 : Received 503 (Service Unavailable) error response for page. http://sussexchef.com/wp-content/uploads/2013/08/HotForkBuffetDelivererd.pdf141N/A50301N/A 503 : Received 503 (Service Unavailable) error response for page. http://sussexchef.com/Afternoon%20Tea.pdf141N/A50301N/A 503 : Received 503 (Service Unavailable) error response for page. http://sussexchef.com/wp-content/uploads/2013/08/Christmas-Promo.pdf141N/A50301N/A 500 : Received 500 (Internal Server Error) error response for page. http://sussexchef.com/?attachment_id=51110N/A50000N/A 503 : Received 503 (Service Unavailable) error response for page. http://sussexchef.com/wp-content/uploads/2013/08/Biography.pdf10N/A50300N/A 503 : Received 503 (Service Unavailable) error response for page. http://sussexchef.com/?attachment_id=62810N/A50300N/A 503 : Received 503 (Service Unavailable) error response for page. http://sussexchef.com/wp-content/uploads/2014/02/2014-02-01-11.12.00.jpg
Getting Started | | SussexChef830 -
'not a valid url' error in campaign set up
I get the error not a valid url when I'm trying to set up a campaign. I know it's a valid url. I have tried with www, non-www, http://, https:// when I do the https it lets me start, but then I get an error that https is forwarding to http and I need to use that. When I then put in the http, I get the original error. thanks in advance for your help.
Getting Started | | HighVoltage0 -
High Number of Crawl Errors for Blog
Hello All, We have been having an issue with very high crawl errors on websites that contain blogs. Here is a screenshot of one of the sites we are dealing with: http://cl.ly/image/0i2Q2O100p2v . Looking through the links that are turning up in the crawl errors, the majority of them (roughly 90%) are auto-generated by the blog's system. This includes category/tag links, archived links, etc. A few examples being: http://www.mysite.com/2004/10/ http://www.mysite.com/2004/10/17/ http://www.mysite.com/tagname As far as I know (please correct me if I'm wrong!), search engines will not penalize you for things like this that appear on auto-generated pages. Also, even if search engines did penalize you, I do not believe we can make a unique meta tag for auto-generate pages. Regardless, our client is very concerned seeing these high number of errors in the reports, even though we have explained the situation to him. Would anyone have any suggestions on how to either 1) tell Moz to ignore these types of errors or 2) adjust the website so that these errors now longer appear in the reports? Thanks so much! Rebecca
Getting Started | | Level2Designs0