I'm getting a Crawl error 605 Page Banned by robots.txt, X-Robots-Tag HTTP Header, or Meta Robots Tag
-
The website is www.bigbluem.com and is a wordpress site.
I'm getting the following error:
605 Page Banned by robots.txt, X-Robots-Tag HTTP Header, or Meta Robots Tag
But what is weird is the domain it lists below that is http://None/BigBlueM.com
Any advice?
-
I can now resolve the www version of the site but not reach the root domain which will continue to return the 605 error so there is something about the root domain configuration that is blocking our bot. A workaround would be to create a new campaign for www.bigbluem.com instead of bigbluem.com
-
Thanks David!
I noticed that this morning it was showing the correct domain all of a sudden. Thanks for looking into that further.
I've made that change to the robots.txt file so go ahead and test when you can.
Thanks again!
-
Hello!
Sorry for the confusion. For your site there were two issues, one on our side with our crawler failing between August 21st - 29th crawls trying to reach http://None/yourdomain.com and the site not responding to robots.txt
The crawl issue has been resolved but for some reason your site is still blocking our user-agent.
Rogerbot can't follow allow directives well so you could try updating it with Disallow:
User-agent: Rogerbot
Disallow:Let me know if this helps! Once you make the change I can run a quick test to see if it will resolve.
-
Thanks!
I've made that change, although I still don't know why it would have the URL as http://None/BigBlueM.comThat's concerning and makes me think that it isn't crawling because it's trying to crawl that URL which doesn't exist.
Do you know if I have to wait another week for Moz to attempt a crawl or can I force that to see if it's working?
-
Thanks for piping in here! I will definitely rely more heavily on GWMT and will check out Screaming Frog SEO spider. Thanks!
-
I've consistently experienced problems with the Moz crawler - to the point of I no longer put much value into it.
I'm getting this error and nothing has changed in my robots.txt.
Instead, use GWMT and Screaming Frog SEO spider - that's all you need and does more than the Moz crawler.
-
I have noticed underneath rogerbot you have dissallow change it to
User-agent: rogerbot
Allow: /
Then let me know how you get on crawling the home page.
-
No problem, I will have a look into that issue for you now it does seem strange.
-
Hey! Thanks for the response.
I didn't have Disallow set up for the root folder at all, just for /wp-admin/.
I went ahead and added the User-agent: rogerbot
The one thing I am still concerned about is that Moz is saying "We were unable to access your homepage" and then has the URL http://None/BigBlueM.com
Why does it think that is my homepage? That seems weird and it isn't that way on any of my other sites that are set up in Moz.
-
In order to allow crawlers to to access the site, you would need to either remove the / after Disallow or change Disallow to Allow.
If you specifically want to allow the Moz crawler, you can insert the following directive above the current directive that is in the .htaccess
User-agent: rogerbot
Disallow: -
If you happen to figure out a solution before someone posts here, let me know what it is!
-
I am having the exact same problem, however Google webmaster tools is able to crawl the site just fine.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Crawl Issue Question
Hey guys, I have run the crawl on my WordPress site and Moz finds a "Critical crawl issue" for my site on a broken link (404 error): mydomain.com/**%25s **, I can't seem to be able to find such a link anyway and I have run the website through several other tools that scan for broken links and such and there is no such result.
Moz Bar | | K.Net
This link doesn't exist on my site at all and I don't know where Moz got it from, I have made changes to my site and recrawled several times and the specific error persists. Does anyone have any ideas?0 -
Moz claims we have meta noindex but we don't
Hi, I'm encountering an issue where moz scan says we have meta noindex, but I have confirmed across several of our pages that this simply isn't true. I have confirmation that the below tag is present: name="robots" content="index, follow" /> I also verified our https header through https://www.webconfs.com/http-header-check.php and see nothing indicating that we are sending any no index headers. Why would the crawler report this when it doesn't seem to be the case? Let me know if I need to provide more information.
Moz Bar | | charper_floqast0 -
Why page load time is different in google webmaster vs what is displayed in moz?
When I analyze the site through Moz tool and compare the results with google webmaster, I am not able to figure what why Moz does not report the slow pages. Fro example this page has an avrage LCP of 3.0 sec https://www.collegehippo.com/graduate-school/programs/gre-score-business-analytics-data-analytics When I see the report in moz, it does not point to any such issue. Should I be worried about what google reports and try to fix the page?
Moz Bar | | etattva0 -
Why do i get a difficulty and CTR score in keyword explorer when there is no min or max monthly volume on a search term?
why do i get a difficulty and CTR score in keyword explorer when there is no min or max monthly volume on a search term?
Moz Bar | | eezysee0 -
How To Stop Moz Crawl From Prepending /blog/ on all our site urls that it crawls
Hello, At some time in the past our WP site had urls like this: www.oursite.com/blog/post-title-pretty-link The site has not used that url structure for quite some time, but Moz crawl is still hitting every post with /blog/prepended and as a result is generating thousands of 404s. When the /blog/ is removed from the url, then the urls work fine. Where are those old urls being stored and how can we update them? How do we address this issue? Any assistance will be appreciated. Thanks!
Moz Bar | | dbcooper1 -
Https address has different result that http in Page Optimization Score toll in Moz PRO
The following url
Moz Bar | | TrueluxGroup
https://www.whichledlight.com/t/gu10-led-bulbs has (100 score for keyword 'GU10 LED')
has different on page opmisation score results to
http://www.whichledlight.com/t/gu10-led-bulbs (73 score for keyword 'GU10 LED') Anyone know if we've set something up wrong?
Also, is this even something to worry about, does google treat them differently? We're using the Page Optimisation Tool in Moz Pro ** UPDATE ** It's worth mentioning we are using emberjs, so the website is a single page application.
We use prerender to render the pages for google.0 -
MOZ crawler 404 errors on wordpress
Hi all, I've got hundreds of issues coming up on the MOZ crawler with 404 errors, I don't know what these URL's are. Here's a couple of examples; http://www.theswagbagco.co.uk/category/watford/http%3A%2F%2Fwww.theswagbagco.co.uk%2F2015%2F10%2F15%2Fnew-products-2%2F
Moz Bar | | vaineh
http://www.theswagbagco.co.uk/2015/10/01/thank-you-epsom/http%3A%2F%2Fwww.theswagbagco.co.uk%2F2015%2F10%2F01%2Fthank-you-epsom%2F See the first one is one page with a different url appended, the second is the same thank-you-epsom url. How would I find out where these are even being linked from?0 -
Perplexed by last MOZ crawling duplicate content errors
In the last crawler issues report from MOZ I can see many many pages listed as duplicate content with 0 duplicate urls. Like this: http://imgur.com/fbikRVq I am puzzled, what does it mean?
Moz Bar | | max.favilli0