When I crawl my site On Moz it says it can't access the robots.txt file, but crawl is fine on SEM Rush - Anyone know any reason for this?
-
Hi guys,
When I try to run a site crawl on Moz it returns an error saying that it has failed due to an error with the robots.txt file.
However, my site can be crawled by SEM Rush with no mention of problems with roots.txt file issues.
My developer has looked into it and insists their is no problem with my robots.txt and I've tried the Moz crawl at least 6 times over an 8 week period.
Has anyone ever seen such a large discrepancy between Moz and SEM Rush or have any ideas why Moz has this issue with my site??
TIA everyone
-
Hi there!
Thanks so much for the great question! I'm so sorry you're having trouble getting your site crawled. Without knowing the exact site you're working with I can't say for sure what's going on but I can offer some suggestions that may help. If you're seeing the crawl is successful with other tools, there may be a bot-specific setting or directive that is banning our crawler, rogerbot, from your site. This may be coming directly from your server or it may be listed in your robots.txt file.
I have a troubleshooting guide here that may help- it outlines some other common issues that can keep rogerbot from being able to move forward with the crawl.
If you're still having trouble, please feel free to send an email on over to help@moz.com with the site you're having trouble with. That way we can take a look and see what's going on!
-
I also had this issue today on my site. Haven’t tried Semrush but got the same error that you are describing when I used Moz
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Has anyone purchased the MOZ SEO courses? Are they good?
I'm looking to learn more about SEO. Has anyone purchased the courses available through MOZ? If so, are they useful?
Getting Started | | Stevepair0 -
Moz only crawling one page of a campaign, please help
Today I set up a new campaign for a client, however the crawl has only found the home page and is saying that the URL is unavailable. The site is definitely live and the URL is correct. I have set up the campaign 3 times one with the full address (http://www.) one with www. and with just the domain name. All three of these have come page with one page crawled and "unavailable" above the URL. It is picking up the crawl issues on the page and showing domain authority but I don't know why it's not crawling other pages. Prior to setting up the campaign I did a site crawl and Moz found everything then, so I don't know why it isn't now. Please help. Thanks
Getting Started | | Wrapped0 -
Moz Pro Warning: Redirect Chain
I have just signed up to a Moz Pro account, and after it finished crawling my website it gave me a warning about a redirection chain. http://elementpaints.com >> https://elementpIaints.com >> https://www.elementpaints.com I'm trying to find some more information about how to fix this problem but I'm not having much luck. This article I found even says it is not a problem: https://really-simple-ssl.com/knowledge-base/avoid-landing-page-redirects So now I'm even more confused, do I still need to fix this? If so, how do I do that? FYI: I have Lets Encrypt SSL cert installed on my server, and I'm using Cloudflare with Full SSL option and HSTS enabled, and "Always Use HTTPS" option is turned on. | http://elementpaints.com |
Getting Started | | elementpaints0 -
Moz Pro for simple local seo status reporting
I have a new client on a small budget with a local yoga studio. There's been no SEO work on this site so far, so, after the basics, I plan to work through the items on this list: https://moz.com/blog/top-20-local-search-ranking-factors-an-illustrated-guide I'm a new Moz Pro user and am looking for advice on how to use Moz Pro tools with my client to review progress over time...a simple checklist of reports/metrics to share and track, and to use in weekly status reports. Or, in other words, how can I configure and use Moz Pro tools for basic progress tracking? This is not a sophisticated, SEO-savvy client, so I want to keep things very simple. Thanks for any ideas or pointers. Dave
Getting Started | | dgbruns0 -
Moz profile
Hi, For some reason when I try and add my site's link in my Moz profile I keep getting an error. My site an SSL so I used https instead of http. I tried again without the 's' as my domain redirects to the https version, but this didn't work. Then I tried without http or https and this didn't work either. Why does Moz hate my site? It's actually really good!!! Thanks, Amelia
Getting Started | | CommT0 -
How do I interpret Duplicate Content in a Crawl Report, when it only gives me a URL? How do I know what is duplicated on that page somewhere else?
I need help interpreting the Crawl Report for Duplicate Content. It gives me the URLs of pages that have duplicate content, but how do I know what content exactly is duplicated elsewhere? And how do I figure out where it is duplicated? Also, are there Moz Analytics articles or videos teaching you how to use each component of the analytics programs? Thanks!
Getting Started | | NancyBryan0 -
Moz's official stance on Subdomain vs Subfolder - does it need updating?
Hi, I am drawing your attention to Moz's Domain basics here: http://moz.com/learn/seo/domain It reads: "Since search engines keep different metrics for domains than they do subdomains, it is recommended that webmasters place link-worthy content like blogs in subfolders rather than subdomains. (i.e. www.example.com/blog/ rather than blog.example.com) The notable exceptions to this are language-specific websites. (i.e., en.example.com for the English version of the website)." I am wondering if this is still Moz's current recommendation on the subfolders vs subdomains debate, given that the above (sort of) implies that SE's may not combine ranking factors to the domain as a whole if subdomains are used - which (sort of) contradicts Matt Cutts last video on the matter ( http://www.youtube.com/watch?v=_MswMYk05tk ) which implies that this is not the case and there is so little difference that their recommendation is to use whatever is easiest. It would also seem to me that if you were looking through the eyes of Google, it would be silly to treat them differently if there were no difference at all other than subdomain vs subfolder as one of the main reasons a user would use a sud-domain is a technical on for which it would not make sense for Google to treat differently in terms of its algorithm. I notice that in terms of Moz, while most of the site uses subfolders, you do have http://devblog.moz.com/ - and I was wondering if this is due to a technical reason or conscious decision, as it would seem to me that the content within this section is indeed linkworthy (as it has external links pointing to it from external sources), therefore it would seem to not be following the initial advice that is posted in Moz's basics on domains. Therefore I am assuming it is due to a technical reason - or that Moz's adive is out of date with current Moz thinking, and is indeed in line with Matt C in that it doesn't matter. Cheers
Getting Started | | James773 -
Can I use wildcards "*" when setting up a new Moz campaign?
Basically I would like the Moz crawler to focus on a specific section of our domain. We do not bucket things via folder groups, so the use of wildcards would be applicable to us. Our URL structure: www.domain.com/some-stuff-here/p12345 Is the example below a valid input to track the above URL structure? www.domain.com//p Thanks.
Getting Started | | WEB-IRS0