Moz site crawl doesn't work
-
The Moz site crawl isn't working for my campaign, but works for the site's on demand crawl. The search should not be disallowed by robots.txt or the headers. I'd like to be able to track the website for the campaign so I can see SEO gains / losses and increases / decreases in indexing.
-
Hey guys,
It does give an error:
**Moz was unable to crawl your site on May 3, 2019. **Our crawler was banned by a page on your site, either through your robots.txt, the X-Robots-Tag HTTP header, or the meta robots tag. Update these tags to allow your page and the rest of your site to be crawled. If this error is found on any page on your site, it prevents our crawler (and some search engines) from crawling the rest of your site. Typically errors like this should be investigated and fixed by the site webmaster. Read our troubleshooting guide.
With that in mind, it does not appear that the crawl should be blocked with the resources in place.
-
Hi there,
Sam from Moz's Help Team here - could you please pop a message about this over to help@moz.com along with the affected campaign name so we can look into this for you?
Thank you!
-
Does it give a warning/error, or is it simply returning no pages?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
We're in Australia are webinars offered during our time
The hours offered for the Moz training are not in our business hours, they're inconvenient.
Getting Started | | RepositPower0 -
What is the IP address of the MOZ crawlbot?
Our preproduction site is restricted to a specific IP range. Prelaunch I'd like to use MOZ to check the site but this obviously isn't currently possible. Is there an IP range I can whitelist ?
Getting Started | | mrphome0 -
What is 'domain authority'?
IN seomoz, it mentions domain authority but it doesnt define it. what does domain authority mean?
Getting Started | | torbett0 -
Why is analytics.moz.com showing up a traffic source in Google Analytics?
Under Traffic Sources, analytics.moz.com is listed as a source for 37 sessions this past month. What might this be? Is it legitimate traffic or should I be filtering this?
Getting Started | | mphdavidson0 -
MOZ Guru Needed for Freelance Work
Hello -- I'm new to Moz and wanted to know if any community members were available for freelance work to help me fine-tune the setup of the Moz account for one of my clients. This could lead to additional work for other clients of my advertising agency.
Getting Started | | KRT0 -
Trying to connect Google Analytics account But it's not listed in the drop down box
I'm trying to connect a Google Analytics account but when I go through the drop down box with our list of clients it doesn't show up. It is listed in our agency Analytics account. We have 30+ accounts on GA. Any reason why the account wouldn't make it into the list? Thanks, ~Anjolie
Getting Started | | RKD_Group0 -
How long after sign up does the on page grader take to work?
Everything I enter comes back with a grade F and 0s across all spectrums. I know this isn't the case so cannot understand it.
Getting Started | | Swifty19810 -
I can't export reports in pdf
Hi,I try to export pdf report for my client but this function doesn't work. I tried in different browsers and on different machines - still doesn't work. What am I doing wrong? Thanks in advance,JJ
Getting Started | | jjtech0