Cannot Crawl ... 612 : Page banned by error response for robots.txt.
-
I tried to crawl www.cartronix.com and I get this error:
612 : Page banned by error response for robots.txt.
I have a robots.txt file and it does not appear to be blocking anything
Also, Search Console is showing "allowed" in the robots.txt test...
I've crawled many of our other sites that are similarly set up without issue.
What could the problem be?
-
Thank you everyone... I'm learning! And you are helping!
-
Great - just checked the robots.txt with web-sniffer & shows a 200 status now so crawl shouldn't be an issue.
Dirk
-
I think I figured it out... For some reason, robots.txt was set at 600...I changed it to 644... I will run crawl again... Thanks.
-
Thank you for the responses. Can you give me any direction on how to correct this? I am lost
-
Your robots.txt renders in a browser - but from technical perspective it generates a 403: Forbidden (check http://www.cartronix.com/robots.txt with web-sniffer.net)
Moz will not crawl if your robots.txt is returning a 403 (see answer from Chiaryn Miranda / Moz on https://moz.com/community/q/without-robots-txt-no-crawling
Quote: "The only commands from the http responses that we consider to block our crawler from accessing a site would be a 403: Forbidden error or a 5xx error."
Dirk
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Cant figture out why On-Page Grader is not recognizing the heading tag text
Hi all, I'm using the On-Page Grader tool to optimize some of our pages: https://analytics.moz.com/pro/research/page-grader?keyword=Russian Immigration Lawyers in Sacramento&url=https%3A%2F%2Fwww.russianlawyers.us%2Fus%2Fcalifornia%2Fsacramento-immigration One of the suggestions is "Optimal Use of Keywords in Header Tags". They keyword is "Russian Immigration Lawyers in Sacramento" and we do have it in an tag on the page (located in the right sidebar). The tool doesnt seem to catch that though. Can someone help me please? Thanks
Moz Bar | | heritageweb1 -
If we put the disavow links in google, does MOZ crawl the same links?
I have put bad or spam links in disavow file, but still showing in MOZ backlinks. So, I want to know that Why is MOZ not removing the spam links from their system?
Moz Bar | | insidewebanalytics0 -
Moz Crawler fails on the first page
Hi guys, Can anybody shed some light, I'm running a crawl on a client's website but it's failing on the homepage and not crawling any other pages. It appears to be throwing an 804: HTTPS (SSL) error and then terminating the crawl. Now, the page in question was serving up mixed content up until about 4 days ago, but has since been fixed. I read that we should wait at least 48 hours before initiating another crawl to avoid hitting a cached version - which I did, but it still appears to be having issues. Is there anything specific I can do to get around this issue? I'm on a trial account and this feature is one I'm keen to test, so there is a bit of a time constraint. Any help is greatly appreciated! Thanks in advance!
Moz Bar | | philipdanielhayton0 -
Internal Links Count in Crawl Report
My understanding of the 'Internal Links' results in a moz crawl report is that it represents the number of links on the given page that link to other pages on the same site.Assuming this is a correct assumption: We recently ran a crawl report on www.phase1tech.com. Some of the pages are coming back with a large amount of 'internal links'. These 2 pages for example are showing 800 internal links: http://www.phase1tech.com/Upcoming-Events
Moz Bar | | AISEO
http://www.phase1tech.com/Contact Then there are a number of pages coming back with 705 Internal Links, including: http://www.phase1tech.com/Dalsa-CameraLink-Cameras
http://www.phase1tech.com/Hitachi-CameraLink-Cameras At best there are approximately 70-80 links on these pages. Where are these large counts coming from? Is there a means to see what the links being reported on are? At the same time the 'Too Many On-Page Links' indicates 'No' for some pages with a high number of links, and 'Yes' for pages with a low number of links. For example: http://www.phase1tech.com/Baumer-SX-Series
Too Many On-Page Links: Yes
Internal Links: 2
What's up with that?0 -
On Page Grader - We have keyword in Meta Description But does not show up
The on-page grader is not counting/picking up the keyword in the meta description. We have double checked the meta description and re-uploaded the page, and checked the source code on-line. The keyword is there in meta description, but not showing up in on-page-grader. https://www.trustdeedrealestateinvesting.com/trimarkfunding/buy-first-trust-deeds.html keyword buy first trust deeds
Moz Bar | | Manifestation0 -
SEO MOZ ERROR
Hello moz comunity, I tried to use the moz keyword difficulty service in the last 2 days and I get this error over and over again... see photo: http://www.evernote.com/shard/s238/sh/5775a179-1be7-4e76-8563-cf087c37cf2b/576bda1a72f446a8806a0f1914193829 Oops Gosh! It looks like something has gone a bit wrong. Don't worry though, we know and are fixing it. How Can I solve this? I need to check a lot of keywords for my websites. Any alternatives? Thank you !!!
Moz Bar | | Sebastyan220 -
My 301 Error and Duplicate Title Content Issue is Growing !
When i redirect some of my page - it shows error. not redirecting and i made this 3-4 months before, no effect. All Errors under each category make me gone sick.
Moz Bar | | Esaky0 -
Moz crawl sees meta description but there are none
I have a new site I have run a starter crawl on. The crawl came back saying some of the pages do indeed have meta descriptions. When I go to the same page and use the Moz Chrome toolbar it says they do not have meta descriptions. I also know they do not have meta descriptions. Are there any instances in which the Moz crawler would see them when they are not there?
Moz Bar | | SBXMedia0