Cannot Crawl ... 612 : Page banned by error response for robots.txt.
-
I tried to crawl www.cartronix.com and I get this error:
612 : Page banned by error response for robots.txt.
I have a robots.txt file and it does not appear to be blocking anything
Also, Search Console is showing "allowed" in the robots.txt test...
I've crawled many of our other sites that are similarly set up without issue.
What could the problem be?
-
Thank you everyone... I'm learning! And you are helping!
-
Great - just checked the robots.txt with web-sniffer & shows a 200 status now so crawl shouldn't be an issue.
Dirk
-
I think I figured it out... For some reason, robots.txt was set at 600...I changed it to 644... I will run crawl again... Thanks.
-
Thank you for the responses. Can you give me any direction on how to correct this? I am lost
-
Your robots.txt renders in a browser - but from technical perspective it generates a 403: Forbidden (check http://www.cartronix.com/robots.txt with web-sniffer.net)
Moz will not crawl if your robots.txt is returning a 403 (see answer from Chiaryn Miranda / Moz on https://moz.com/community/q/without-robots-txt-no-crawling
Quote: "The only commands from the http responses that we consider to block our crawler from accessing a site would be a 403: Forbidden error or a 5xx error."
Dirk
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
I have recently switzed to SSL on ramxpert.dk, now Moz is not able to crawl the site anymore
Hi Moz I have recently switzed to SSL on ramxpert.dk, now Moz is not able to crawl anymore. When I look at the crawl report there is just one fault displayed: "403 : Received 403 (Forbidden) error response for page." How can i solve this issue asap? I was not aware that there would be any issues with moz when activating a SSL certificate on the site. The campaign with the problem is: ramxpert.dk. Regards
Moz Bar | | WebBoost
Jesper Nielsen0 -
Rankings > Landing Page > Only UK?
Hello Is it possible to use the landing Page reports for sites in Norway .no? If we define a search engine in UK, maybe it will mess up all the other data we are collecting? Regards
Moz Bar | | moggiew
Mogens Stoltz Wennersten0 -
How can I find duplicate pages from a Moz Crawl?
We have many duplicate pages that show up on the Moz Crawl, and we're trying to fix these but it's very difficult because I can't see a way to isolate the code where the duplicate is found. For instance, http://experiencemission.org/immersion/ is one of our main pages, and the crawl shows one duplicate of http://experiencemission.org/immersion. It appears that one of our staff manually edited the source code in one of our pages but forgot the trailing slash. This would be an easy fix but the problem is that this page is linked to internally on our website 2423 times, so it's next to impossible to find the code that is incorrect. We have many other pages with this same basic problem. We know we have duplicates, but it's next to impossible to isolate them. So my question is this: When viewing the Moz Crawl data is there any way to see where a specific duplicate page link is located on our website? Thanks for any and all help!
Moz Bar | | expmission0 -
Crawl Test Takes Long Time
Hi Moz, I have submitted our website for a crawl test. Usually it would only take a few hours to do the crawl. However this time, it takes quite long time and the result still shows in progress 😞 This is a small website which only contains less than 10 pages. Just wondering if this is our website setting issue or it is a technical issue at your end? Many thanks in advance. sFjAERG.png
Moz Bar | | russellbrown0 -
500 errors showing up differently on moz and google wmt
Lately, I've been having the issue of a large increase in 500 errors. These errors seem to be intermittent, in other words, Google and Moz are showing that I have server 500 errors for many pages but, when I actually check the links, everything's fine. I've run tests to see if there is any virus on the server or if I have any corrupt files and as far as I can tell, there are none. I'm left with the possibility that maybe one of my plugins is causing this issue (I'm built on top of Wordpress). Moz is showing that I had nearly five hundred 500 server errors on the 12th or the 11th. On the other hand, Google shows that on the 13th I had 179 server errors and then an additional 200 for the 15th. I'm assuming Google is slow to find or report these things? I would like to know which is more reliable so that I can try to figure out which of these plugins may be causing the problem, if any or if I'm investigating this the wrong way, I'd love to have more suggestions. Thanks in advance! Sorry, the url is http://www.heartspm.com if you'd like to take a look.
Moz Bar | | GerryWeitz0 -
On Page Grader Changed?
Hi, Could anyone tell me if the on page grader has changed? I remember seeing 8 boxes at the top but now i'm only seeing 6... Before I remember seeing a H2-H4 and a Strong/B Box? Thanks
Moz Bar | | dmxsta0 -
Confusing Moz Crawl?
Hi there, I am not sure if I am missing on something but the moz crawls are rather confusing. After singing in I have received 11 emails with crawls and today I have received again new, When I go to check there to the dashboard it shows 26 pages with issues. When I scroll down I see the pages with issue. Then when I click on the first page listed, to view the issues it says this: Rel Canonical
Moz Bar | | Rebeca1
Using rel=canonical suggests to search engines which URL should be seen as canonical. For this site: http://villasdiani.com/ but we have sorted out the canonical issues a long time ago. Is this a wrong information or is it really true that we do not specify the canonical for our site? Then the second page with issue is there listed http://villasdiani.com/beach-villas/ and it says: Duplicate Page Title
You should use unique titles for your different pages to ensure that they describe each page uniquely and don't compete with each other for keyword relevance. But it does not point out which page is duplicate with this one! I do not have any other page named the same way. It also says in Issues overview 26pages with issues, but it shows on the bottom only 5 under and when I click on view more it brings me to high priority issues where is 0. The most is freaking me out this report: When I click on links, there are listed on the bottom the pages with highest authority among which I found this http://villasdiani.com/db I have never created this kind of page! Funny enough when I click on it it really open that page! How this can be??? In issues overview it also shows on the bottom, right corner 11 page with duplicate content but when I click on it to review it it brings me to high priority issues windows where is not displayed anything Can somebody advice me regarding of this. I have sign up here to learn and sort out the problems with the site but so far I am only getting more confused here. Thank you very much for looking into this.0 -
Is Manual Crawl Test option available now to Pro Users?
Hi all, I have worked on my Crawl Issues and want to see how many still exist. Earlier I was using Manual Crawl Test. However, now I don't see this tool in Moz Account. Please suggest. Thanks
Moz Bar | | chandman0