I'm getting a Crawl error 605 Page Banned by robots.txt, X-Robots-Tag HTTP Header, or Meta Robots Tag
-
The website is www.bigbluem.com and is a wordpress site.
I'm getting the following error:
605 Page Banned by robots.txt, X-Robots-Tag HTTP Header, or Meta Robots Tag
But what is weird is the domain it lists below that is http://None/BigBlueM.com
Any advice?
-
I can now resolve the www version of the site but not reach the root domain which will continue to return the 605 error so there is something about the root domain configuration that is blocking our bot. A workaround would be to create a new campaign for www.bigbluem.com instead of bigbluem.com
-
Thanks David!
I noticed that this morning it was showing the correct domain all of a sudden. Thanks for looking into that further.
I've made that change to the robots.txt file so go ahead and test when you can.
Thanks again!
-
Hello!
Sorry for the confusion. For your site there were two issues, one on our side with our crawler failing between August 21st - 29th crawls trying to reach http://None/yourdomain.com and the site not responding to robots.txt
The crawl issue has been resolved but for some reason your site is still blocking our user-agent.
Rogerbot can't follow allow directives well so you could try updating it with Disallow:
User-agent: Rogerbot
Disallow:Let me know if this helps! Once you make the change I can run a quick test to see if it will resolve.
-
Thanks!
I've made that change, although I still don't know why it would have the URL as http://None/BigBlueM.comThat's concerning and makes me think that it isn't crawling because it's trying to crawl that URL which doesn't exist.
Do you know if I have to wait another week for Moz to attempt a crawl or can I force that to see if it's working?
-
Thanks for piping in here! I will definitely rely more heavily on GWMT and will check out Screaming Frog SEO spider. Thanks!
-
I've consistently experienced problems with the Moz crawler - to the point of I no longer put much value into it.
I'm getting this error and nothing has changed in my robots.txt.
Instead, use GWMT and Screaming Frog SEO spider - that's all you need and does more than the Moz crawler.
-
I have noticed underneath rogerbot you have dissallow change it to
User-agent: rogerbot
Allow: /
Then let me know how you get on crawling the home page.
-
No problem, I will have a look into that issue for you now it does seem strange.
-
Hey! Thanks for the response.
I didn't have Disallow set up for the root folder at all, just for /wp-admin/.
I went ahead and added the User-agent: rogerbot
The one thing I am still concerned about is that Moz is saying "We were unable to access your homepage" and then has the URL http://None/BigBlueM.com
Why does it think that is my homepage? That seems weird and it isn't that way on any of my other sites that are set up in Moz.
-
In order to allow crawlers to to access the site, you would need to either remove the / after Disallow or change Disallow to Allow.
If you specifically want to allow the Moz crawler, you can insert the following directive above the current directive that is in the .htaccess
User-agent: rogerbot
Disallow: -
If you happen to figure out a solution before someone posts here, let me know what it is!
-
I am having the exact same problem, however Google webmaster tools is able to crawl the site just fine.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
I have too many tittle tag issue for my site on moz site crawl error
I have too many tittle tag issue in site crawl error but when I checked manually for the error there is no title in source code. Please Help me to understand
Moz Bar | | Nileshaggarwal0 -
How does Moz determine the number of keywords evident on a given page
How does Moz identify which keywords are evident on page and how many that can be on that page. Can any of these identified keywords lie on any other pages within site and to what degree-thanks
Moz Bar | | bickybeanie0 -
On-Page Grader URL inaccessible when copy/pasted but not when edited
Hi!, I've looked through multiple topics on this but none quite seem to fit what's going on - hopefully someone can help! I get the error message 'Sorry, but that URL is inaccessible.' when I copy and paste a url from my site into the search e.g. http://www.orbussoftware.com/enterprise-architecture/ However if I edit this to https the search completes fine. Since we redesigned our site approx 6 months ago, we've found most of our rankings have completely dropped off, and now I'm getting this error I'm wondering if it has something to do with how our site is structured? If I'm getting this error with Moz does that mean Google could be having issues too? Or is it all just a strange quirk? Thanks!
Moz Bar | | JennaOrbus0 -
Why do i get multiple variations of my url with ?order=asc and ?view=list at the end of it in my crawl report?
I just did a crawl for one my clients to validate any error in the structure. Next thing I know is that the website have multiple variation of the same url with query like ?order=asc and ?view=list at the end of it. I am wondering why these url variations appears in the crawl I just did since bots aren't suppose to go further thant the ? normally. Just to show you a couple of url's of my crawl test. <colgroup><col width="484"></colgroup>
Moz Bar | | alexrbrg
| https://test.com/exemple/?per_page=9 |
| https://test.com/exemple/?per_page=15 |
| https://test.com/exemple/?per_page=30 |
| https://test.com/exemple/?orderby=popularity |
| https://test.com/exemple/?orderby=date |
| https://test.com/exemple/?orderby=price |
| https://test.com/exemple/?orderby=price-desc |
| https://test.com/exemple/?order=asc |
| https://test.com/exemple/?order=desc |
| https://test.com/exemple/?view=list | Thank you Guys0 -
On Page Grader Help
Hi There I am going through the process of using the on page grader and optimising lots of pages, it seems that the difference between getting an F and A grade is simply the title having those keywords however there are keywords that I want to grade well for without stuffing the Title tag with loads of different keywords. Would you say the on page grader is reliable given this and if not could it be optimised to consider the other factors rather then just displaying a tick to say these factors have been considered and achieved.
Moz Bar | | sharpleaddesign0 -
Moz crawl issues: All pages keep resolving to our "cookies not enabled" page
Upon running the Moz Pro site crawler, I noticed that I received quite a bit of duplicate titles along with 302 redirects (which is our site creating a temporary 302 to our "cookies not enabled" page). How would I get around the crawler being redirected to this page? I've never ran across this issue before, despite using the crawler with sites that use the same framework as the one thats affected. Any ideas?
Moz Bar | | responsivelabs0 -
How Can I intreptret The Crawl Report Resulst?
Hello, I am new to Moz and I have received 2 crawl reports. The first one was ok. I made a few changes to my site plugins, and my next crawl report came up with 41 4XX errors. Basically, a lot of my posts. I went back to my plugins and saw the following plugins: 404 redirect plugin & Utlimate Tiny MCE I reactivated both. I am presuming that these must have caused the issues or maybe my site was hacked. I re ran a crawl this morning, but I don't know what the different headings mean or how to understand the report. Can anyone advise? My site is new and just started to go up the rankings...so quite disappointed with this set back. regards Chriss
Moz Bar | | chrisspell0 -
SEO MOZ ERROR
Hello moz comunity, I tried to use the moz keyword difficulty service in the last 2 days and I get this error over and over again... see photo: http://www.evernote.com/shard/s238/sh/5775a179-1be7-4e76-8563-cf087c37cf2b/576bda1a72f446a8806a0f1914193829 Oops Gosh! It looks like something has gone a bit wrong. Don't worry though, we know and are fixing it. How Can I solve this? I need to check a lot of keywords for my websites. Any alternatives? Thank you !!!
Moz Bar | | Sebastyan220