Cannot Crawl ... 612 : Page banned by error response for robots.txt.
-
I tried to crawl www.cartronix.com and I get this error:
612 : Page banned by error response for robots.txt.
I have a robots.txt file and it does not appear to be blocking anything
Also, Search Console is showing "allowed" in the robots.txt test...
I've crawled many of our other sites that are similarly set up without issue.
What could the problem be?
-
Thank you everyone... I'm learning! And you are helping!
-
Great - just checked the robots.txt with web-sniffer & shows a 200 status now so crawl shouldn't be an issue.
Dirk
-
I think I figured it out... For some reason, robots.txt was set at 600...I changed it to 644... I will run crawl again... Thanks.
-
Thank you for the responses. Can you give me any direction on how to correct this? I am lost
-
Your robots.txt renders in a browser - but from technical perspective it generates a 403: Forbidden (check http://www.cartronix.com/robots.txt with web-sniffer.net)
Moz will not crawl if your robots.txt is returning a 403 (see answer from Chiaryn Miranda / Moz on https://moz.com/community/q/without-robots-txt-no-crawling
Quote: "The only commands from the http responses that we consider to block our crawler from accessing a site would be a 403: Forbidden error or a 5xx error."
Dirk
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz Crawl only crawling the top level page (1 page)
For the past few mounts my weekly site crawl has been inconsistent. One week works fine, it crawls all of my 500 or so pages. The following week it only crawls 1 page (http://mydomain.com) and nothing else. A few weekly scan go by and the crawl is back up the the 500 or so pages.I went ahead and created several campaigns with duplicate settings and crawled the site. Most times but not all the new campaign's crawl works fine crawling all pages. But within a week or two the weekly crawl will fail again. (crawling 1 page). Currently i have four campaign's all with the same settings running weekly crawls. 2 campaign's crawled the 500 pages and two crawled only the single page. Any help will be greatly appreciated
Moz Bar | | dmaude0 -
Site crawl only shows homepage
Hi everyone, A client of us has a quite new website with a lot of URLs. (Google Search Console indicates around 5300.) However, when I execute a site crawl with screaming frog, or a crawl test in MOZ, it only shows me one URL, the homepage. Does somebody have an idea why the other pages of the website are not showing up? Thanks,
Moz Bar | | WeAreDigital_BE
Jens0 -
Moz Crawler Causing Server Timeouts... Crawling thousands of non-existant pages with query parameters
Moz crawler is crawling all pages like this: http://www.xxxx.com/?product_count=100&product_order=desc&product_orderby=date http://www.xxxx.com/?product_count=100&product_order=desc&paged=1 http://www.xxx.com/?product_count=100&product_order=desc&product_view=grid Last month it crawled 80,000 pages on a site with less than 100 pages. Is there a way to select only certain pages to be crawled? Right now it is still crawling this site, since Monday morning and it's Tuesday mid-day. Every Monday it is causing time-outs from high band width on our server. Just getting ready to delete this client from the account unless there is a solution someone can give us. Thanks.
Moz Bar | | adirondack0 -
Page title not being read
Hi all I have been testing my site with the Moz on page grader recently and every page seems to come up with the same issues. The keyword doesn't seem to be in the title tag or the title tag isn't there. Could anyone tell me why this is? This is an example of a page it is happening on https://www.sbat.com/tips/todays-free-horse-racing-tips I used several versions of the Keyword including free horse Racing tips Horse racing tips and it seemed none of these were picked up.
Moz Bar | | gillies88880 -
What are the best tools to help analyse on page optimisation for pages on development server and not currently live
currently using seo quake and moz tool bar but wondered if there is a better suggestion that will look at pages that are only accessible on the internal network on development server. Very restricted in what can be installed
Moz Bar | | Dan-Moz0 -
Error 605
Hi I have been getting the ollowing error on my dashboard the last 6-8 weeks error 605: Page Banned by robots.txt, X-Robots-Tag HTTP Header, or Meta Robots Tag It does seem to be able to crawl the site so what does this mean? The only issue I see is that I don't get any crawl errors listed. Could this be a prob with moz? I had this site checked on a different account on moz and there didnt get the error! The site is http://www.copyfaxes.com Thanks
Moz Bar | | henya0 -
Duplicate Page Title query in the PRO Campaign tool
Can someone help me on this. I am seeing duplicate page titles on the PRO Campaign Crawl tool on an ecommerce site for example MOZ is saying that these two pages have a duplicate page title: http://www.cheapsnapframes.co.uk/colour-25mm-snap-frames/25mm-green-snap-frame/a0-traffic-green-snap-frame-25mm/prod_1730.html http://www.cheapsnapframes.co.uk/snap-picture-poster-frames/colour-25mm-snap-frames/green-25mm-snap-frame/a0-traffic-green-snap-frame-25mm/prod_1730.html They are the the same product in two categories. When I view the source of both pages the this link is the same in the meta: <link rel="<a class="attribute-value">canonical</a>" href="[http://www.cheapsnapframes.co.uk/colour-25mm-snap-frames/25mm-green-snap-frame/a0-traffic-green-snap-frame-25mm/prod_1730.html](view-source:http://www.cheapsnapframes.co.uk/colour-25mm-snap-frames/25mm-green-snap-frame/a0-traffic-green-snap-frame-25mm/prod_1730.html)" /> So is there something else I need to have done to erradicate this or is it not an issue? Thanks in advance Tracy
Moz Bar | | dashesndots0 -
Not getting foreign characters in crawl diagnostics .csv
The crawl diagnostics .csv file is showing high-ascii characters instead of the correct language (foreign language website) e.g. Vietnamese, Chinese (both kinds), etc. Is there a way to get this right?
Moz Bar | | trainSEM0