When I crawl my site On Moz it says it can't access the robots.txt file, but crawl is fine on SEM Rush - Anyone know any reason for this?
-
Hi guys,
When I try to run a site crawl on Moz it returns an error saying that it has failed due to an error with the robots.txt file.
However, my site can be crawled by SEM Rush with no mention of problems with roots.txt file issues.
My developer has looked into it and insists their is no problem with my robots.txt and I've tried the Moz crawl at least 6 times over an 8 week period.
Has anyone ever seen such a large discrepancy between Moz and SEM Rush or have any ideas why Moz has this issue with my site??
TIA everyone
-
Hi there!
Thanks so much for the great question! I'm so sorry you're having trouble getting your site crawled. Without knowing the exact site you're working with I can't say for sure what's going on but I can offer some suggestions that may help. If you're seeing the crawl is successful with other tools, there may be a bot-specific setting or directive that is banning our crawler, rogerbot, from your site. This may be coming directly from your server or it may be listed in your robots.txt file.
I have a troubleshooting guide here that may help- it outlines some other common issues that can keep rogerbot from being able to move forward with the crawl.
If you're still having trouble, please feel free to send an email on over to help@moz.com with the site you're having trouble with. That way we can take a look and see what's going on!
-
I also had this issue today on my site. Haven’t tried Semrush but got the same error that you are describing when I used Moz
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved What would the exact text be for robots.txt to stop Moz crawling a subdomain?
I need Moz to stop crawling a subdomain of my site, and am just checking what the exact text should be in the file to do this. I assume it would be: User-agent: Moz
Getting Started | | Simon-Plan
Disallow: / But just checking so I can tell the agency who will apply it, to avoid paying for their time with the incorrect text! Many thanks.0 -
New to SEO? Check out Moz Academy!
Easy-to-access SEO education has always been dear to our mission. Simply log in with your Moz Profile and start learning. Whether you're new to SEO or looking to take your skills to the next level, Moz Academy has fun, affordable courses for the entire team! Check it out now! moz-academy-training.png
Getting Started | | HayleyBowyer2 -
Where can I find Moz Rank for my websites?
I am unable to find an option in Moz Pro where I can find Moz rank for my website. Please help me to find that.
Getting Started | | TapanSant1 -
Where can I find the list of all the question I've asked here?
In my profile I see only the comments, I'd like to keep track also of my questions
Getting Started | | 2mlab0 -
Clarifications on the Moz Analytics package (Medium - $149 per month)
What are the Moz tools available with this package? What factors of SEO can be checked with these tools? With this package, is it possible to provide a single URL (preferably home page) and Moz will analyse the entire site and highlight how the site performs wrt various SEO factors? This package states that with this package we can run 10 Moz Analytics campaigns. Our understanding of Moz Analytics Campaign is every site; say www.test.com is one analytics campaign. Are we correct? Does the subdomains within a parent domain also considered as one analytics campaign. For e.g., if I have sites: www.mydomain.com and www.xxx.mydomain.com are they considered two separate campaigns or are they considered as one single campaign? In this package it is listed as 750 keywords, what does this signify? In what way this feature can be used to check our site’s SEO compliance. Please elaborate. In this package it is listed as 15 social accounts, what does this signify? In what way this feature can be used to check our site’s SEO compliance. Please elaborate. What do you mean by branded reports?
Getting Started | | WebCCTrial0 -
Moz health check for newbies
Hi I'm on the free trial, I'm finding it really useful. since I was 16 (16years ago) all I've done is cook. After 3 weeks of Moz I feel like I'm or the way to marketing my business instead of having others tell me how to do it. A while back I used to use SEO freelancers to buff my site, but recently I realized that I was not doing enough to promote my business myself, I was trying to pay others to do it for me. I goggled Soe and marketing, of course I found Moz while doing so! The content is exceptional and really goes in depth and we are now coming up with a new content strategy! The problem is that I have this gut feeling that The moz product is worth the money it asks for as I can easily identify the issues or opportunities my site has, I still feel that I lack the competence to use moz to its full potential! Is there an option to have someone skpe me and give me some tips or in sites with in my own campaign? I've spent a lot of time reading the blog content and watching the videos but this all seams to focus on how to get results but not so much on how to use the Moz console/dashboard. I've probably just missed a hole bunch of content but if not do Moz have and option to help set up your Moz analytic and Research so you can get the most relevant info to you? Any help is most welcome Kind regards Ben Horle SussexChef.com
Getting Started | | SussexChef831 -
Moz Staff: History + Watch List
Hi All, I have been going crazy trying to find a thread I read last week ... I think!
Getting Started | | Mark_Ch
Several search attempts later, no closer to finding the thread in question. A genius moment or simple frustration got me thinking. How cool would it be to have the following features on MOZ Q&A Forum. Watch List
1] Check off any threads that you would like to monitor over time. Should you stumble upon a thread of value then simply add to watch list and view it's progress over time. No searching for the thread in question. History
2] Keep a history off all threads viewed over a period of time. Thanks Mark0 -
Custom reports in Moz Analytics?
I am trying to find where to configure whitelabled custom reports in moz analytics. Where can I find them? If they are only accessible through the old version, how do I switch back?
Getting Started | | keybroker1