I'm getting a Crawl error 605 Page Banned by robots.txt, X-Robots-Tag HTTP Header, or Meta Robots Tag
-
The website is www.bigbluem.com and is a wordpress site.
I'm getting the following error:
605 Page Banned by robots.txt, X-Robots-Tag HTTP Header, or Meta Robots Tag
But what is weird is the domain it lists below that is http://None/BigBlueM.com
Any advice?
-
I can now resolve the www version of the site but not reach the root domain which will continue to return the 605 error so there is something about the root domain configuration that is blocking our bot. A workaround would be to create a new campaign for www.bigbluem.com instead of bigbluem.com
-
Thanks David!
I noticed that this morning it was showing the correct domain all of a sudden. Thanks for looking into that further.
I've made that change to the robots.txt file so go ahead and test when you can.
Thanks again!
-
Hello!
Sorry for the confusion. For your site there were two issues, one on our side with our crawler failing between August 21st - 29th crawls trying to reach http://None/yourdomain.com and the site not responding to robots.txt
The crawl issue has been resolved but for some reason your site is still blocking our user-agent.
Rogerbot can't follow allow directives well so you could try updating it with Disallow:
User-agent: Rogerbot
Disallow:Let me know if this helps! Once you make the change I can run a quick test to see if it will resolve.
-
Thanks!
I've made that change, although I still don't know why it would have the URL as http://None/BigBlueM.comThat's concerning and makes me think that it isn't crawling because it's trying to crawl that URL which doesn't exist.
Do you know if I have to wait another week for Moz to attempt a crawl or can I force that to see if it's working?
-
Thanks for piping in here! I will definitely rely more heavily on GWMT and will check out Screaming Frog SEO spider. Thanks!
-
I've consistently experienced problems with the Moz crawler - to the point of I no longer put much value into it.
I'm getting this error and nothing has changed in my robots.txt.
Instead, use GWMT and Screaming Frog SEO spider - that's all you need and does more than the Moz crawler.
-
I have noticed underneath rogerbot you have dissallow change it to
User-agent: rogerbot
Allow: /
Then let me know how you get on crawling the home page.
-
No problem, I will have a look into that issue for you now it does seem strange.
-
Hey! Thanks for the response.
I didn't have Disallow set up for the root folder at all, just for /wp-admin/.
I went ahead and added the User-agent: rogerbot
The one thing I am still concerned about is that Moz is saying "We were unable to access your homepage" and then has the URL http://None/BigBlueM.com
Why does it think that is my homepage? That seems weird and it isn't that way on any of my other sites that are set up in Moz.
-
In order to allow crawlers to to access the site, you would need to either remove the / after Disallow or change Disallow to Allow.
If you specifically want to allow the Moz crawler, you can insert the following directive above the current directive that is in the .htaccess
User-agent: rogerbot
Disallow: -
If you happen to figure out a solution before someone posts here, let me know what it is!
-
I am having the exact same problem, however Google webmaster tools is able to crawl the site just fine.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Monthly Keyword Volume Differences Keyword Research V's Campaign Rankings
Why are there differences in the keyword volume data, is this a UK only issue? Am I missing something? Within Keyword Research;
Moz Bar | | GrouchyKids
suspended ceiling systems swindon (0-10) Within Campaign Rankings:
suspended ceiling systems swindon (no data) It's not just one instance either its across multiple keywords.0 -
Page Optimization Error
Hi, I am trying to track a page optimization feature for one of my project, https://www.writemyessay247.com for keyword: write my essay but i keep getting this below error: "Page Optimization Error There was a problem loading this page. Please make sure the page is loading properly and that our user-agent, rogerbot, is not blocked from accessing this page." I checked robots.txt file, it all looks fine. Not sure what is the problem? Is it a problem with Moz or the website?
Moz Bar | | j.rahimi19900 -
On-Page Grader supporting HTTPS?
Hi there, I am getting the error "Sorry, but this URL is inaccessible" on the Moz On-Page Grader. I have tested with multiple URL's but any that link to my site give the error. I have noticed my site's URL's redirect to https so my question is..is the on-page grader in-compatible with https? Cheers!
Moz Bar | | ToteIreland
Ross0 -
MOZ On Page Grader Not Seeing Changes
Hello all, I've been doing some SEO work recently and using the really handy A-F rating on MOZ on page grader as a guide. It's worked great so far on my sites up until today. I've made some small changes to a site such as the title and MOZ isn't seeing these changes. For example if I were to put 'banana' in the title of one of my other sites and re-run the on page grader with 'banana' as a keyword, instantly MOZ would say this keyword is used in the title once. I've done the same to another site and the same thing hasn't happened. I did a test and wrote all the keywords from the title in the on page grader and it still said they appear 0 times in the title. Am I missing something here? The site is different from the others, it's flat html and not CMS, but the SEO changes I've made are still the same. Thanks!
Moz Bar | | HB170 -
Why my on-page report card won't improve?
For my page www.homedestination.com/Minneapolis-homes-for-sale.html, Moz ranks the page as an "F" for the keyword "Minneapolis home buyers". All data in the report card is from months ago. using the "Grade my On-Page Optimization" button fails to show any updates - even after months. For example the title is actually "Minneapolis Homes For Sale| buying a Minneapolis home | Homebuyers" while Moz says it is ""Minneapolis Homes For Sale | Home Destination | Jenna Thuening" Moz says the description tag is "Realtor Jenna Theuning helping real estate buyers find Minnetonka homes for sale, owner of Home Destination. CDPE also helps with foreclosure and short sales buying." and "Minneapolis homes for sale, and Minneapolis listing, and Minneapolis upscale real estate, and Minneapolis lakeside properties provided by Home Destination"" when it is actually "Minneapolis homes for sale, and Minneapolis listing, Minneapolis home buyers, and Minneapolis lakeside properties provided by Home Destination." Additionally, my weekly ranking reports seems "stuck" and continues to show many "F" ranked keywords that have been updated for some time. Amy advice on how to see improvements?
Moz Bar | | jessential1 -
408 errors in crawl diagnostics
Best community, The Crawl Diagnostics Report of Moz gave our website a lot of 408 errors like below: <dl> <dt>Title</dt> <dd>408 : Error</dd> <dt>Meta Description</dt> <dd>408 Request Time-out</dd> <dt>Meta Robots</dt> <dd>Not present/empty</dd> <dt>Meta Refresh</dt> <dd>Not present/empty</dd> <dd>-----------------------------------------------------------------------</dd> <dd>The report has diagnosed a lot of these (around 320), even though we cannot reproduce the error (we cannot seem to find it ourself). </dd> <dd>2 questions relating to this: </dd> <dd>* Can you (the people of Moz) reproduce the errors manually? </dd> <dd>* Is it possible that it is a bug in the spider of Moz itself (too many spiders crawling at the same time)?</dd> </dl>
Moz Bar | | arjen.koedam0 -
Is Manual Crawl Test option available now to Pro Users?
Hi all, I have worked on my Crawl Issues and want to see how many still exist. Earlier I was using Manual Crawl Test. However, now I don't see this tool in Moz Account. Please suggest. Thanks
Moz Bar | | chandman0 -
Why isn't seo moz properly crawling my site?
In my campaign results, only 2 pages have been crawled, when there are many more. Also, when I do the on-page evaluation for my homepage and the main keyword, I get a grade of F, with a ton of errors that my page actually does not have. It's as if SEO moz is having trouble crawling through my site. Any ideas on this?
Moz Bar | | diplomajim0