Moz not able to crawl our site - any advice?
-
When I try and crawl our site through Moz it gives this message:
Moz was unable to crawl your site on Aug 7, 2019. Our crawler was banned by a page on your site, either through your robots.txt, the X-Robots-Tag HTTP header, or the meta robots tag. Update these tags to allow your page and the rest of your site to be crawled. If this error is found on any page on your site, it prevents our crawler (and some search engines) from crawling the rest of your site. Typically errors like this should be investigated and fixed by the site webmaster.
I have been through all the help and doesn't seem to be any issues. You can check the site and robots.txt here: https://myfamilyclub.co.uk/robots.txt.
Anyone got any advice on where I could go to get this sorted?
-
Hello, have the same issue and MOZ getting "Bummer no data found" for domain https://wherebuycoin.com/ . The SEMRush and Ahrefs see this site, but MOZ not, even after 2 years. What can be wrong? Where to read why MOZ not crawling my site?
Thanks!
-
Check your website's robots.txt file to ensure that it's not blocking Moz's user-agent or specific directories/pages that you want to be crawled. Update the robots.txt file if necessary to allow Moz to crawl your site.
-
Hey,
We crawled the site just fine with Screaming Frog. Perhaps this is a Moz issue?
Looks like they are well aware of it though.
Thanks!
-
Hi,
I'm sorry to hear about this! Would you be able to write into help@moz.com as this will allows us to investigate into this via the most convenient method.
Looking forward to hearing from you,
Eli
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Risk of moz on website
Does Moz collect information of customers accessing my website https://tructiepbongda.site/?
Getting Started | | gogoanimetp
Thank you!0 -
Where can I find Moz Rank for my websites?
I am unable to find an option in Moz Pro where I can find Moz rank for my website. Please help me to find that.
Getting Started | | TapanSant1 -
Moz can't crawl my site.
Moz cannot carry out the site crawl on my online shop. Not really sure what the issue is, it has no problem getting onto my site when you use www. before the address, but it needs to be able to access bluerinsevintage.co.uk Stuck as what to do, we are a shopify store. Anyone else had this problem, or know what i need to change so they can crawl the site? thjis is the page they are getting when trying to get on bluerinsevintage.co.uk but if they use www.bluerinsevintage.co.uk the site comes up. Adam
Getting Started | | bluerinsevintage0 -
Crawl rate
How often does Moz crawl my website ? (I have a number of issues I believe I have fixed, and wondered if there was a manual request to re-crawl ?) Thanks. Austin.
Getting Started | | FuelDump0 -
Can not create new campain with my site: edunet.com.vn
I can not create new campain because it always warning my site is not a right URL. I don't understand, please tell what should I do. My site is: edunet.com.vn. (When I try to use "Grade a page for keyword" for URL: edunet.com.vn or edunet.com.vn/thong-tin-tuyen-sinh, it returns "Sorry, but that URL is inaccessible".) Thank you so much. Minh Tam.
Getting Started | | toosol0 -
Custom reports in Moz Analytics?
I am trying to find where to configure whitelabled custom reports in moz analytics. Where can I find them? If they are only accessible through the old version, how do I switch back?
Getting Started | | keybroker1 -
Lost since the change over to Moz
I was recently lead to a series of videos which looked to be quite helpful here http://moz.com/help/pro/introduction But is seems with the change over from SEO Moz to Moz, nothing is in the same place. Should I bother watching them or wait for new ones? I'm particularly interested in SEO to raise our rank.
Getting Started | | AliciaMarie0