Moz not able to crawl our site - any advice?
-
When I try and crawl our site through Moz it gives this message:
Moz was unable to crawl your site on Aug 7, 2019. Our crawler was banned by a page on your site, either through your robots.txt, the X-Robots-Tag HTTP header, or the meta robots tag. Update these tags to allow your page and the rest of your site to be crawled. If this error is found on any page on your site, it prevents our crawler (and some search engines) from crawling the rest of your site. Typically errors like this should be investigated and fixed by the site webmaster.
I have been through all the help and doesn't seem to be any issues. You can check the site and robots.txt here: https://myfamilyclub.co.uk/robots.txt.
Anyone got any advice on where I could go to get this sorted?
-
Hello, have the same issue and MOZ getting "Bummer no data found" for domain https://wherebuycoin.com/ . The SEMRush and Ahrefs see this site, but MOZ not, even after 2 years. What can be wrong? Where to read why MOZ not crawling my site?
Thanks!
-
Check your website's robots.txt file to ensure that it's not blocking Moz's user-agent or specific directories/pages that you want to be crawled. Update the robots.txt file if necessary to allow Moz to crawl your site.
-
Hey,
We crawled the site just fine with Screaming Frog. Perhaps this is a Moz issue?
Looks like they are well aware of it though.
Thanks!
-
Hi,
I'm sorry to hear about this! Would you be able to write into help@moz.com as this will allows us to investigate into this via the most convenient method.
Looking forward to hearing from you,
Eli
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz DA Issue
Hey Community, This is my site which is the Best Drifitng in Australia. I am getting a problem and my site's DA is unable to show using. I want to see it as soon as possible, although I am using a premium tool. Thanks
Getting Started | | jndjue40 -
What does Location National means in Moz Pro Campaigns?
I understand the concept of locations in Moz Pro Campaigns. What does National exactly means? Is it like globally for a United States Google Search Rank or what? Please explain.
Getting Started | | ds9.tech1 -
Moz only crawling one page of a campaign, please help
Today I set up a new campaign for a client, however the crawl has only found the home page and is saying that the URL is unavailable. The site is definitely live and the URL is correct. I have set up the campaign 3 times one with the full address (http://www.) one with www. and with just the domain name. All three of these have come page with one page crawled and "unavailable" above the URL. It is picking up the crawl issues on the page and showing domain authority but I don't know why it's not crawling other pages. Prior to setting up the campaign I did a site crawl and Moz found everything then, so I don't know why it isn't now. Please help. Thanks
Getting Started | | Wrapped0 -
Crawl issues, how to see a referring link?
Hi There, We've got two crawl issues for pages that don't exist (and never existed). The links are strange and judging by the code in them, appear to be coming from our own CMS. How can we see which pages the links are on in Moz? Cheers Ben
Getting Started | | cmscss0 -
Open site explorer
Excuse my ignorance but I am very new to all this. I have a new site www.sassandgrace.co.uk and am using MOZ to try and get it right first time. On OSE I am seeing different results for http://www.sassandgrace.co.uk/ and http://www.sassandgrace.co.uk i.e with an without / ) and different results again for http://sassandgrace.co.uk These differences relate to the social page metrics but I would be keen to get one consistent reading. What do I need to do? Also, I know that I have links to the site but none are showing at all. On google console there are a couple showing but I know that there are many more. Is it just a matter of time before they are 'found' or is there something that I should be doing? Sorry for what are probably very basic questions but any help is appreciated.
Getting Started | | Sassandgrace0 -
Moz Analytics Shows isights, but none when i click
Hi, i have a campaign report that shows 8 Insights, however when i click on that link which says 8 Insights, the page is blank. Any info would be appreciated. Thanks
Getting Started | | andrewlos0 -
Whenever I try to access campaigns in moz pro I get an error page
I recently signed-up for a new pro account. As I was adding my first subdomain everything was fine until I was asked to link to GA, when I clicked yes I got this error message: 403 Forbidden Now every time I click on set-up campaign I get taken to a page with nothing but the 403 Forbidden text.
Getting Started | | Toptal0