I'm getting a Crawl error 605 Page Banned by robots.txt, X-Robots-Tag HTTP Header, or Meta Robots Tag
-
The website is www.bigbluem.com and is a wordpress site.
I'm getting the following error:
605 Page Banned by robots.txt, X-Robots-Tag HTTP Header, or Meta Robots Tag
But what is weird is the domain it lists below that is http://None/BigBlueM.com
Any advice?
-
I can now resolve the www version of the site but not reach the root domain which will continue to return the 605 error so there is something about the root domain configuration that is blocking our bot. A workaround would be to create a new campaign for www.bigbluem.com instead of bigbluem.com
-
Thanks David!
I noticed that this morning it was showing the correct domain all of a sudden. Thanks for looking into that further.
I've made that change to the robots.txt file so go ahead and test when you can.
Thanks again!
-
Hello!
Sorry for the confusion. For your site there were two issues, one on our side with our crawler failing between August 21st - 29th crawls trying to reach http://None/yourdomain.com and the site not responding to robots.txt
The crawl issue has been resolved but for some reason your site is still blocking our user-agent.
Rogerbot can't follow allow directives well so you could try updating it with Disallow:
User-agent: Rogerbot
Disallow:Let me know if this helps! Once you make the change I can run a quick test to see if it will resolve.
-
Thanks!
I've made that change, although I still don't know why it would have the URL as http://None/BigBlueM.comThat's concerning and makes me think that it isn't crawling because it's trying to crawl that URL which doesn't exist.
Do you know if I have to wait another week for Moz to attempt a crawl or can I force that to see if it's working?
-
Thanks for piping in here! I will definitely rely more heavily on GWMT and will check out Screaming Frog SEO spider. Thanks!
-
I've consistently experienced problems with the Moz crawler - to the point of I no longer put much value into it.
I'm getting this error and nothing has changed in my robots.txt.
Instead, use GWMT and Screaming Frog SEO spider - that's all you need and does more than the Moz crawler.
-
I have noticed underneath rogerbot you have dissallow change it to
User-agent: rogerbot
Allow: /
Then let me know how you get on crawling the home page.
-
No problem, I will have a look into that issue for you now it does seem strange.
-
Hey! Thanks for the response.
I didn't have Disallow set up for the root folder at all, just for /wp-admin/.
I went ahead and added the User-agent: rogerbot
The one thing I am still concerned about is that Moz is saying "We were unable to access your homepage" and then has the URL http://None/BigBlueM.com
Why does it think that is my homepage? That seems weird and it isn't that way on any of my other sites that are set up in Moz.
-
In order to allow crawlers to to access the site, you would need to either remove the / after Disallow or change Disallow to Allow.
If you specifically want to allow the Moz crawler, you can insert the following directive above the current directive that is in the .htaccess
User-agent: rogerbot
Disallow: -
If you happen to figure out a solution before someone posts here, let me know what it is!
-
I am having the exact same problem, however Google webmaster tools is able to crawl the site just fine.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Crawler triggering Spam Throttle and creating 4xx errors
Hey Folks, We have a client with an experience I want to ask about. The Moz crawler is showing 4xx errors. These are happening because the crawler is triggering my client's spam throttling. They could increase from 240 to 480 page loads per minute but this could open the door for spam as well. Any thoughts on how to proceed? Thanks! Kirk
Moz Bar | | kbates1 -
Page Grader states "includes Canonical Tag" but it's not in the page source at all
I've ran it multiple times and changed other things it picked up on so not sure where it's getting the canonical tag is included even though it isn't?
Moz Bar | | Wana-Ryd0 -
How to find a list of pages with missing H1 tags
An external SEO/PPC agency did an audit of our site a little while back and said that over 10% of our pages were missing an H1 tag. I am trying to find a way to gather a full list of these in order for our web company to fix. I downloaded the Crawl Test report thinking it would include info on tags in there but it doesn't seem to. Is there a different tool I can use that will get me this information?
Moz Bar | | Lepra0 -
Can't delete items from the on page grader
I check every single box and they don't delete. This is driving me nuts. Please can you delete them for me because I am not impressed with this AT ALL. In fact I am getting so cross I am in danger of screaming hysterically which might get me the sack and it would be all your fault. That was slightly tongue in cheek, but please can you fix it please. please.
Moz Bar | | CommT0 -
On Page Grader Returning Large # of Keywords
When using the on-page grader the results show the below for a page with a specific keyword in the body only 5 times : We found this keyword used 1100 times. Any Idea why this would be showing such a high number? The keywords are in Thai language but there is a space before and after the keyword. Thanks.
Moz Bar | | brettjohn670 -
How can the Moz Page Grader support a 'keyword portfolio' approach?
I used to use the Page Grader tools to support the old philosophy of one page - one keyword. With more focus now being given to a portfolio of keywords around a topic area - what would be a good approach to using the page grader tool? Obviously getting A's and B's is impossible for multiple keywords. The only way i've seen suggested in moz tools to help with keyword portfolios is to use labels in the ranking measurement and then find averages of the results. Are there other strategies that I can try?
Moz Bar | | AISFM0 -
Can't get on page grader to work properly.
Hi I'm trying to optimize my pages with the on page grader tool but it keeps returning an F grade when I know my page is very well optimized. It is like something is blocking the page crawl but I have double checked my robots.txt and can't think of anything else that is causing a problem. I am trying to do www.hydrohobby.co.uk with the keyword hydroponics for starters but it is the same problem with all other page urls on my site and keywords I try to input. This is a new site made with cscart 4.0. I've graded pages on previous versions of this software with no problems. Can anyone help? Rob
Moz Bar | | hydrohobby0 -
Issues with on-page report
My on page reports are really bad (F´s all over the place). When I go to a specific page to check why, it seems that Moz is not properly tracking some of the key metrics. For example for the following page (https://www.iguanafix.com.ar/servicios/pisos/palermo?loc=AR.CF.CF.PA) and keyword "pisos palermo" it is tracking that the keyword is not in the title (which it is), the keyword is not in the URL (which clearly is), not in H1, again it is, etc. I guess this is what is affecting my ranking and thus preventing me from defining a strategy to improve my SEO. Any ideas much appreciated. Thanks
Moz Bar | | IguanaFix0