Moz not able to crawl our site - any advice?
-
When I try and crawl our site through Moz it gives this message:
Moz was unable to crawl your site on Aug 7, 2019. Our crawler was banned by a page on your site, either through your robots.txt, the X-Robots-Tag HTTP header, or the meta robots tag. Update these tags to allow your page and the rest of your site to be crawled. If this error is found on any page on your site, it prevents our crawler (and some search engines) from crawling the rest of your site. Typically errors like this should be investigated and fixed by the site webmaster.
I have been through all the help and doesn't seem to be any issues. You can check the site and robots.txt here: https://myfamilyclub.co.uk/robots.txt.
Anyone got any advice on where I could go to get this sorted?
-
Hello, have the same issue and MOZ getting "Bummer no data found" for domain https://wherebuycoin.com/ . The SEMRush and Ahrefs see this site, but MOZ not, even after 2 years. What can be wrong? Where to read why MOZ not crawling my site?
Thanks!
-
Check your website's robots.txt file to ensure that it's not blocking Moz's user-agent or specific directories/pages that you want to be crawled. Update the robots.txt file if necessary to allow Moz to crawl your site.
-
Hey,
We crawled the site just fine with Screaming Frog. Perhaps this is a Moz issue?
Looks like they are well aware of it though.
Thanks!
-
Hi,
I'm sorry to hear about this! Would you be able to write into help@moz.com as this will allows us to investigate into this via the most convenient method.
Looking forward to hearing from you,
Eli
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
New to SEO? Check out Moz Academy!
Easy-to-access SEO education has always been dear to our mission. Simply log in with your Moz Profile and start learning. Whether you're new to SEO or looking to take your skills to the next level, Moz Academy has fun, affordable courses for the entire team! Check it out now! moz-academy-training.png
Getting Started | | HayleyBowyer2 -
I want to increase the DA of my Site
Dear all i have established a new site and want to increase its DA fast. So i need your suggestions in this regard. This newly established blog is my first independant project so i want to make it perfect.
Getting Started | | hamza52520 -
How to rank website on google with help of Moz ?
I am new here and my question is how rank a website with moz because my website was not ranking in any keywords so plz guide me
Getting Started | | RoyDigitalWorld0 -
We recently switched from HTTP to HTTPS and we are having crawling issues!
We switched our website from HTTP to HTTPS and we started to get an email from Moz about the robots.txt being unable to crawl our website. The website is hosted through wordpress but we haven't had any issues until we switched. We have no idea what to do or even what the problem is! If you have had a similar problem and fixed it, we need your help! Thank you.
Getting Started | | DrInfinity0 -
What is the IP address of the MOZ crawlbot?
Our preproduction site is restricted to a specific IP range. Prelaunch I'd like to use MOZ to check the site but this obviously isn't currently possible. Is there an IP range I can whitelist ?
Getting Started | | mrphome0 -
MOZ Monthly Custom Report Question
How do I set the Moz to produce monthly report on 1st of every month. I dont see any option there. we have 50+ Clients in Moz now and we need to generate report on 1st day of all month to send them report. Its 3rd Oct today and I don't see my previous month report being generated. Can you guys help me
Getting Started | | emarketexperts0 -
Site traffic vs other sites
I have a quick question. Does Moz have a tool I can use to see how much traffic my site gets a month versus some competition? I know Alexa has this but you have to create an account. I am just wondering if Moz has one and I am missing it.
Getting Started | | trumpfinc0 -
Why does moz show "not in top 50" for all my keywords???
Hello, I signed up to moz pro 4 days ago. And so far it seems to be tracking visits etc. But all my keywords say "not in top 50" . Why is this? Is this normal? Just to confirm most of the keywords i pasted in from my webmaster tools and i only chose the ones that were in top 50
Getting Started | | casper09030