Moz not able to crawl our site - any advice?
-
When I try and crawl our site through Moz it gives this message:
Moz was unable to crawl your site on Aug 7, 2019. Our crawler was banned by a page on your site, either through your robots.txt, the X-Robots-Tag HTTP header, or the meta robots tag. Update these tags to allow your page and the rest of your site to be crawled. If this error is found on any page on your site, it prevents our crawler (and some search engines) from crawling the rest of your site. Typically errors like this should be investigated and fixed by the site webmaster.
I have been through all the help and doesn't seem to be any issues. You can check the site and robots.txt here: https://myfamilyclub.co.uk/robots.txt.
Anyone got any advice on where I could go to get this sorted?
-
Hello, have the same issue and MOZ getting "Bummer no data found" for domain https://wherebuycoin.com/ . The SEMRush and Ahrefs see this site, but MOZ not, even after 2 years. What can be wrong? Where to read why MOZ not crawling my site?
Thanks!
-
Check your website's robots.txt file to ensure that it's not blocking Moz's user-agent or specific directories/pages that you want to be crawled. Update the robots.txt file if necessary to allow Moz to crawl your site.
-
Hey,
We crawled the site just fine with Screaming Frog. Perhaps this is a Moz issue?
Looks like they are well aware of it though.
Thanks!
-
Hi,
I'm sorry to hear about this! Would you be able to write into help@moz.com as this will allows us to investigate into this via the most convenient method.
Looking forward to hearing from you,
Eli
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Crawling issue
Hi, I have to set up a campaign for a webshop. This webshop is a subdomain itself. First question: The two subfolders I need to track are /nl_BE and /fr_BE. What is the best way to handle this? Shall I set up two different campaigns for each subfolder, or shall I just make one campaign and add tags to keywords? **Second question: **it seems like Moz can't crawl enough pages. There are no disallows in the robots.txt. Should I try putting the following at the top into my robots.txt? User-agent: rogerbot
Getting Started | | Mat_C
Disallow: Or is it because I want to crawl only a subdomain that it doesn't work? Thanks0 -
My page can not be crawled
Hi all,Hope you could you help me here.I just seen this message but I don't know how to fix it?"Your page redirects or links to a page that is outside of the scope of your campaign settings. Your campaign is limited to pages with _____ in the URL path, which prevents us from crawling through the redirect or the links on your page. To enable a full crawl of your site, you may need to create a new campaign with a broader scope, adjust your redirects, or add links to other pages that include ig.com/de. Typically errors like this should be investigated and fixed by the site webmaster."Any ideas about how should I fix it?Thanks
Getting Started | | lauracelada23100 -
My question is, when you translate your website to another language, does moz crawl both or do i have to add another campaign to moz so that they can crawl it seperately?
Hello, i recently translated my website to spanish, keywords,meta tags, content etc. All of our URL for the translated pages start "ep", which symbolizes espanol. My question is, does moz crawl those pages along with my english pages? or do i have add another campaign for moz to crawl my spanish pages seperately?
Getting Started | | prestigeluxuryrentals.com0 -
MOZ Guru Needed for Freelance Work
Hello -- I'm new to Moz and wanted to know if any community members were available for freelance work to help me fine-tune the setup of the Moz account for one of my clients. This could lead to additional work for other clients of my advertising agency.
Getting Started | | KRT0 -
Can I upgrade my moz subscription from large to premium without getting my campaigns being deleted?Do i need to wait till the end of the month to resubscribe?
I'm planning to take a large subscription of moz..but i may need a premium account in the future.Can I upgrade it in the middle of my current subscription or will i have to wait till my subscription gets over? Also, I'm planning to take only one month currently but i'll renew it once the month completes so can i continue the same account without my campaigns being deleted?
Getting Started | | kdcdmp0 -
High Number of Crawl Errors for Blog
Hello All, We have been having an issue with very high crawl errors on websites that contain blogs. Here is a screenshot of one of the sites we are dealing with: http://cl.ly/image/0i2Q2O100p2v . Looking through the links that are turning up in the crawl errors, the majority of them (roughly 90%) are auto-generated by the blog's system. This includes category/tag links, archived links, etc. A few examples being: http://www.mysite.com/2004/10/ http://www.mysite.com/2004/10/17/ http://www.mysite.com/tagname As far as I know (please correct me if I'm wrong!), search engines will not penalize you for things like this that appear on auto-generated pages. Also, even if search engines did penalize you, I do not believe we can make a unique meta tag for auto-generate pages. Regardless, our client is very concerned seeing these high number of errors in the reports, even though we have explained the situation to him. Would anyone have any suggestions on how to either 1) tell Moz to ignore these types of errors or 2) adjust the website so that these errors now longer appear in the reports? Thanks so much! Rebecca
Getting Started | | Level2Designs0 -
How to remove campaign from Moz.com
I've accidentally created a duplicate campaign, which I am unable to delete, please help
Getting Started | | shopperlocal_DM0