Moz site crawl doesn't work
-
The Moz site crawl isn't working for my campaign, but works for the site's on demand crawl. The search should not be disallowed by robots.txt or the headers. I'd like to be able to track the website for the campaign so I can see SEO gains / losses and increases / decreases in indexing.
-
Hey guys,
It does give an error:
**Moz was unable to crawl your site on May 3, 2019. **Our crawler was banned by a page on your site, either through your robots.txt, the X-Robots-Tag HTTP header, or the meta robots tag. Update these tags to allow your page and the rest of your site to be crawled. If this error is found on any page on your site, it prevents our crawler (and some search engines) from crawling the rest of your site. Typically errors like this should be investigated and fixed by the site webmaster. Read our troubleshooting guide.
With that in mind, it does not appear that the crawl should be blocked with the resources in place.
-
Hi there,
Sam from Moz's Help Team here - could you please pop a message about this over to help@moz.com along with the affected campaign name so we can look into this for you?
Thank you!
-
Does it give a warning/error, or is it simply returning no pages?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Domain Authority hasn't recovered since August
I really need some major advice on this one. Back in September, I asked a question on here as follows: "A client wanted to change their domain name, which we have now done. The site content itself is exactly the same. We put 301 redirect links in so that Google searchers would redirect from the old site to the new one. However Moz then said that it couldn't crawl the old domain because of the redirects and advised creating a brand new campaign for the new domain. We have done this but now Moz says that the domain authority of the new site is 2 (it was 14 on the old domain)." My original question and the answers I got are here: https://moz.com/community/q/new-domain-wipes-out-domain-authority). Generally the responses I got were that we should give Moz time to crawl the new domain and process all the "new" pages. It is now February, ie 6 months after the domain rename, and on Moz the site still has a DA of 2. It seems like 6 months is enough time to wait. We checked all the recommended guides and believe we have done it all correctly. I really don't know what to do now. Can anyone help or have a quick look and work out why this is so bad? Specifics are:
Getting Started | | mfrgolfgti
old domain: https://ryemeadcleaning.co.uk
new domain: https://ryemeadgroup.co.uk0 -
Rel Canonical Notice in Moz Analytics
Hello all. I just received my first report fro Moz Analytics. Signed up here to see what problems my site might have and find out how to fix them. On all of my issue pages I see notice with black dot saying that my urls are Rel Canonical Using rel=canonical suggests to search engines which URL should be seen as canonical. I know that recently my web developer set up so that every url on my website has a rel=canonical as I think there was an issue with trailing slash. Is this to be considered an issue or I can just leave it? Thank you guys. Regards, Armands
Getting Started | | A_Fotografy0 -
How can i start in moz?
I want to know what to do first, how do I start the branding, etc. Thanks!!!
Getting Started | | Gridiron2361 -
Multiple Google Analytics Accounts connected to the same MOZ subscription
Hi everyone, I have a quick question regarding setting up my MOZ account and it's
Getting Started | | Bernardo.Reguero
integration with a Google Analytics account: I'm not sure if I'm right but I think I can only set a single Google
Analytics account to do my analysis and then created several campaign. Or is there a possibility to set several Google Analytics accounts and creating several camapign to track diferente websites (connected to diferente Google Analytics accounts) ? Thanks! Cheers, Bernardo0 -
How long after sign up does the on page grader take to work?
Everything I enter comes back with a grade F and 0s across all spectrums. I know this isn't the case so cannot understand it.
Getting Started | | Swifty19810 -
Crawl Diagnostics
Hello Experts, today i was analyse one of my website with moz and get issue overview and get total 212 issue 37 high all derive to this same url http://blogname.blogspot.com/search?updated-max=2013-10-30T17:59:00%2B05:30&max-results=4&reverse-paginate=true so can anyone help me how to find this url and remove all high priority error. and even on page website get A grade then why not performing well in SE ?
Getting Started | | JulieWhite0 -
MOZ cannot connect to GA
As I went thru the process of setting up my campaign, MOZ could not connect to my GA account. Any idea why?
Getting Started | | selectlocks0