When I crawl my site On Moz it says it can't access the robots.txt file, but crawl is fine on SEM Rush - Anyone know any reason for this?
-
Hi guys,
When I try to run a site crawl on Moz it returns an error saying that it has failed due to an error with the robots.txt file.
However, my site can be crawled by SEM Rush with no mention of problems with roots.txt file issues.
My developer has looked into it and insists their is no problem with my robots.txt and I've tried the Moz crawl at least 6 times over an 8 week period.
Has anyone ever seen such a large discrepancy between Moz and SEM Rush or have any ideas why Moz has this issue with my site??
TIA everyone
-
Hi there!
Thanks so much for the great question! I'm so sorry you're having trouble getting your site crawled. Without knowing the exact site you're working with I can't say for sure what's going on but I can offer some suggestions that may help. If you're seeing the crawl is successful with other tools, there may be a bot-specific setting or directive that is banning our crawler, rogerbot, from your site. This may be coming directly from your server or it may be listed in your robots.txt file.
I have a troubleshooting guide here that may help- it outlines some other common issues that can keep rogerbot from being able to move forward with the crawl.
If you're still having trouble, please feel free to send an email on over to help@moz.com with the site you're having trouble with. That way we can take a look and see what's going on!
-
I also had this issue today on my site. Haven’t tried Semrush but got the same error that you are describing when I used Moz
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How can keyword explorer help me search on a more local level?
I am a total novice at this. I am taking the tutorial and the first thing she addresses is Keyword Explorer. It makes sense to me, but what doesn't is that it asks me to look for keywords in USA. I need to explore keywords on a local level. Anyone out there who can help me with this? am I over my head with Moz Pro if I am a complete novice?
Getting Started | | grettelp1 -
Moz health check for newbies
Hi I'm on the free trial, I'm finding it really useful. since I was 16 (16years ago) all I've done is cook. After 3 weeks of Moz I feel like I'm or the way to marketing my business instead of having others tell me how to do it. A while back I used to use SEO freelancers to buff my site, but recently I realized that I was not doing enough to promote my business myself, I was trying to pay others to do it for me. I goggled Soe and marketing, of course I found Moz while doing so! The content is exceptional and really goes in depth and we are now coming up with a new content strategy! The problem is that I have this gut feeling that The moz product is worth the money it asks for as I can easily identify the issues or opportunities my site has, I still feel that I lack the competence to use moz to its full potential! Is there an option to have someone skpe me and give me some tips or in sites with in my own campaign? I've spent a lot of time reading the blog content and watching the videos but this all seams to focus on how to get results but not so much on how to use the Moz console/dashboard. I've probably just missed a hole bunch of content but if not do Moz have and option to help set up your Moz analytic and Research so you can get the most relevant info to you? Any help is most welcome Kind regards Ben Horle SussexChef.com
Getting Started | | SussexChef831 -
Why cant I add my site to a campaign?
I am trying to add my site www.dominickdalsanto.com to a campaign and it keeps telling me the URL is invalid. I have tried entering it as: dominickdalsanto.com dominickdalsanto.com/ www.dominickdalsanto.com http://www.dominickdalsanto.com Nothing works. I even tried the redirect domains I have ayres-seo.com and still nothing. I tried a few of my other sites too and it works for only one of them. It also would not take dominickinargentina.com Can someone help me please? Thanks!
Getting Started | | Ayres-SEO0 -
When connecting your Google Analytics account to your Moz account, Moz provides a drop-down of accounts & UA codes to select from. Where is Moz pulling those selection options from?
The list of Google Analytics accounts and UA codes listed are outdated. How can I update them to reflect the most up-to-date accounts (& UA codes associated with those accounts)?
Getting Started | | SearchParty0 -
Moz Pro Trial...."Oops! Try refreshing the page"
I am in my MOZ PRO trial. I want to add a campaign to see how I like it, but something keeps coming up saying "Oops! Try refreshing the page, if that doesn't work click here to contact our help team" And I can't find an answer to that question here. Please help ASAP Ld3rHLi
Getting Started | | WFC-Sales0 -
Moz Staff Should Consider it - important
Hello Rand, I was looking for the best content quality checker and I've found many websites saying Free to service. but I got bad experienced there was something poorly coded system on their website so they couldn't check the content quality and duplication. So I suggest you to make a tool that should be helpful for users who are seeking to find out the quality of their content. it should Tell us following factors which are important! Content quality score - English and Grammar Duplication Uniqueness Suggestion to optimize the content
Getting Started | | shubham12340 -
Link Detox or I can use Open Site Explorer for tracking down bad links?
Here's the thing. I need to find bad external links pointing to my site. Is Link Detox the only option or I can actually use Open Site Explorer for that. If OSE is an option, please give me an idea how I need to go about it. Thanks.
Getting Started | | VinceWicks0
