Moz not able to crawl our site - any advice?
-
When I try and crawl our site through Moz it gives this message:
Moz was unable to crawl your site on Aug 7, 2019. Our crawler was banned by a page on your site, either through your robots.txt, the X-Robots-Tag HTTP header, or the meta robots tag. Update these tags to allow your page and the rest of your site to be crawled. If this error is found on any page on your site, it prevents our crawler (and some search engines) from crawling the rest of your site. Typically errors like this should be investigated and fixed by the site webmaster.
I have been through all the help and doesn't seem to be any issues. You can check the site and robots.txt here: https://myfamilyclub.co.uk/robots.txt.
Anyone got any advice on where I could go to get this sorted?
-
Hello, have the same issue and MOZ getting "Bummer no data found" for domain https://wherebuycoin.com/ . The SEMRush and Ahrefs see this site, but MOZ not, even after 2 years. What can be wrong? Where to read why MOZ not crawling my site?
Thanks!
-
Check your website's robots.txt file to ensure that it's not blocking Moz's user-agent or specific directories/pages that you want to be crawled. Update the robots.txt file if necessary to allow Moz to crawl your site.
-
Hey,
We crawled the site just fine with Screaming Frog. Perhaps this is a Moz issue?
Looks like they are well aware of it though.
Thanks!
-
Hi,
I'm sorry to hear about this! Would you be able to write into help@moz.com as this will allows us to investigate into this via the most convenient method.
Looking forward to hearing from you,
Eli
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How do I check PA in moz
i see where I can check the DA for a site, but how can I check the PA of a page?
Getting Started | | Konvertica0 -
Where can I find Moz Rank for my websites?
I am unable to find an option in Moz Pro where I can find Moz rank for my website. Please help me to find that.
Getting Started | | TapanSant1 -
How to have MOZ site crawl pre-launch
Hi, Our new website is about to launch. We would love to have moz.com SITE CRAWL our site before launch. For issues like "missing meta description" and everything else that moz.com checks. We would love to do it before we launch. The new site is currently on a different domain than our live site. example.com <-- this is our live site. new-site.com <-- this is our "staging" server with the new site. We have a long running campaign for example.com Do we need to create a new campain for new-site.com ? Or is there some other simpler way? When we launch we will switch the site from new-site.com to example.com .. example.com will be the address for the new site.. Any ideas or suggestions? best practices? edit Forgot to say thank you for your help and input 🙂
Getting Started | | tandvarden0 -
Moz scraper
How often do you Moz do whatever it is they do for me to get up-to-date data?
Getting Started | | infinety0 -
Moz Analytics Shows isights, but none when i click
Hi, i have a campaign report that shows 8 Insights, however when i click on that link which says 8 Insights, the page is blank. Any info would be appreciated. Thanks
Getting Started | | andrewlos0 -
Moz Pro won't stay connected to my Google Analytics profile
I just started a trial of Moz Pro today and it seems to have a problem staying connected to my Google Analytics account. Evenr time I refresh a page or click to a different page, I get a notification that reads "Our connection to your Google Analytics account was lost. Don't worry, you won't lose any data. Please reauthorize now." I click the link to reauthorize and it says it was successful, But then as soon as I refresh the page or click to a new page withing Moz Pro, the message appears again. I'm afraid that none of my GA data is being collected by Moz Pro. What should I do?
Getting Started | | InterCall0 -
Analytics stats not showing up in MOZ Campaign report
I connected a MOZ campaign to a client's Analytics. It seems it's connected properly with the same Analytics ID # and everything, but the stats still show 0 in MOZ, even though there are thousands of visits in Analytics for the same date range. Pls help! Derek
Getting Started | | DerekM42420