I'm getting a Crawl error 605 Page Banned by robots.txt, X-Robots-Tag HTTP Header, or Meta Robots Tag
-
The website is www.bigbluem.com and is a wordpress site.
I'm getting the following error:
605 Page Banned by robots.txt, X-Robots-Tag HTTP Header, or Meta Robots Tag
But what is weird is the domain it lists below that is http://None/BigBlueM.com
Any advice?
-
I can now resolve the www version of the site but not reach the root domain which will continue to return the 605 error so there is something about the root domain configuration that is blocking our bot. A workaround would be to create a new campaign for www.bigbluem.com instead of bigbluem.com
-
Thanks David!
I noticed that this morning it was showing the correct domain all of a sudden. Thanks for looking into that further.
I've made that change to the robots.txt file so go ahead and test when you can.
Thanks again!
-
Hello!
Sorry for the confusion. For your site there were two issues, one on our side with our crawler failing between August 21st - 29th crawls trying to reach http://None/yourdomain.com and the site not responding to robots.txt
The crawl issue has been resolved but for some reason your site is still blocking our user-agent.
Rogerbot can't follow allow directives well so you could try updating it with Disallow:
User-agent: Rogerbot
Disallow:Let me know if this helps! Once you make the change I can run a quick test to see if it will resolve.
-
Thanks!
I've made that change, although I still don't know why it would have the URL as http://None/BigBlueM.comThat's concerning and makes me think that it isn't crawling because it's trying to crawl that URL which doesn't exist.
Do you know if I have to wait another week for Moz to attempt a crawl or can I force that to see if it's working?
-
Thanks for piping in here! I will definitely rely more heavily on GWMT and will check out Screaming Frog SEO spider. Thanks!
-
I've consistently experienced problems with the Moz crawler - to the point of I no longer put much value into it.
I'm getting this error and nothing has changed in my robots.txt.
Instead, use GWMT and Screaming Frog SEO spider - that's all you need and does more than the Moz crawler.
-
I have noticed underneath rogerbot you have dissallow change it to
User-agent: rogerbot
Allow: /
Then let me know how you get on crawling the home page.
-
No problem, I will have a look into that issue for you now it does seem strange.
-
Hey! Thanks for the response.
I didn't have Disallow set up for the root folder at all, just for /wp-admin/.
I went ahead and added the User-agent: rogerbot
The one thing I am still concerned about is that Moz is saying "We were unable to access your homepage" and then has the URL http://None/BigBlueM.com
Why does it think that is my homepage? That seems weird and it isn't that way on any of my other sites that are set up in Moz.
-
In order to allow crawlers to to access the site, you would need to either remove the / after Disallow or change Disallow to Allow.
If you specifically want to allow the Moz crawler, you can insert the following directive above the current directive that is in the .htaccess
User-agent: rogerbot
Disallow: -
If you happen to figure out a solution before someone posts here, let me know what it is!
-
I am having the exact same problem, however Google webmaster tools is able to crawl the site just fine.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
External links that don't exist
I did a crawl test and it shows that there are 2215 external links on my home page here: https://www.playshakespeare.com An "External Link" is defined here: https://moz.com/researchtools/crawl-test "Number of links on a page that point to a page that is not on the same domain as that page." There are NOT 2215 external links on that page. There are only a handful at any given time. Why is the crawl test reporting thousands over that?
Moz Bar | | playshakespeare0 -
Duplicate Page Titles detected, no relevant links shown
Moz is reporting duplicate page titles, but the relevant pages aren't being shown. The downloaded report doesn't show them either. Screencast: http://screencast.com/t/gkyRds6u
Moz Bar | | ElykInnovation0 -
Keyword used 1448 times ? for a Chinese page
Hi, I tested my page grade here : https://moz.com/researchtools/on_page_grader_reports/1269197 Keyword : "專案管理軟體" , which means "Project Management Software" in Traditional Chinese. URL: http://helpmecloud.com/clarizen-introduction.html But the On Page Grader told me "We found this keyword used 1448 times." I think the result is wrong, I guess you calculated all Chinese letters separately. Am I right ? How can I get a correct grade for my page ? Thansk for help! William
Moz Bar | | Helpmecloud0 -
Is it normal for my homepage's page authority to be higher than my domain authority?
Hi everyone, I've just been checking my MOZ authority scores, and the page authority of my home page, www.concerthotels.com is 53 (mozRank of 6.16) which seems pretty good to me. However, the domain authority is only showing as 45. Is this typical? There is such a big difference between these two scores - what does that suggest? Is there anything obvious that I can do to bring the domain authority up to the level of authority that the homepage commands? Similarly, the UK version of my site has a homepage authority of 41, but the domain authority is only 30. Looking at numerous other websites, quite often the homepage and domain authority scores are very similar, or the domain authority is slightly higher, so it does look as if this is unusual. Any comments would be greatly appreciated, Many thanks Mike
Moz Bar | | mjk260 -
Chrome moz toolbar page analysis not loading
Often the chrome moz toolbar page analysis doesn't load just says Please wait until the page finishes loading.
Moz Bar | | genkee0 -
How can I find the old ERRORS and WARNINGS report in the NEW Moz design?
I'm looking for a complete list of errors and bugs that need to be fixed within a website. I used to use the MAIN tool (at least it seemed it was the most popular) but now that its just MOZ.com I can't seem to find that great report. It had data such as: 1. List of pages with Title Tags too long 2. List of pages with Description Tags too long 3. List of RED errors and YELLOW warnings, BLUE somethings... etc... Ring a bell? I LOVED this report, where can I find this data? Thanks! Derek
Moz Bar | | DerekM42420 -
Is there a way to get Page Authority values included in the Crawl Diagnostic .csv export?
Would be nice to have these values included so that you can sort by Page Authority. 4uF6efx.png
Moz Bar | | WebReputationBuilders0 -
Find all the back links to all the posts/pages within the blog subdirectory only.
Hi, I am new to Moz. I using the open site explorer to find backlinks for a website's blog. The website itself is huge. I want to find all the backlinks to all the posts/pages within the blog subdirectory only. Not the regular website. I ran a few reports, but it is giving me links to that page, not all the sub pages.Fi
Moz Bar | | DarrenD0