Unsolved Moz crawler not crawling on my site
-
Hi all,
im facing an issue where moz crawler is unable to crawl my site. The following error keeps showing Our crawler was banned by a page on your site, either through your robots.txt, the X-Robots-Tag HTTP header, or the meta robots tag.
This is my robots.txt file : https://www.wearefutureheads.com/robots.txt
I'm not sure what else am I missing.. can anyone help
-
@teikh Hi, thanks for contacting us! Would you be able to write into help@moz.com so that our support team can take a closer look at your query please? Thanks
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Crawler was not able to access the robots.txt
I'm trying to setup a campaign for jessicamoraninteriors.com and I keep getting messages that Moz can't crawl the site because it can't access the robots.txt. Not sure why, other crawlers don't seem to have a problem and I can access the robots.txt file from my browser. For some additional info, it's a SquareSpace site and my DNS is handled through Cloudflare. Here's the contents of my robots.txt file: # Squarespace Robots Txt User-agent: GPTBot User-agent: ChatGPT-User User-agent: CCBot User-agent: anthropic-ai User-agent: Google-Extended User-agent: FacebookBot User-agent: Claude-Web User-agent: cohere-ai User-agent: PerplexityBot User-agent: Applebot-Extended User-agent: AdsBot-Google User-agent: AdsBot-Google-Mobile User-agent: AdsBot-Google-Mobile-Apps User-agent: * Disallow: /config Disallow: /search Disallow: /account$ Disallow: /account/ Disallow: /commerce/digital-download/ Disallow: /api/ Allow: /api/ui-extensions/ Disallow: /static/ Disallow:/*?author=* Disallow:/*&author=* Disallow:/*?tag=* Disallow:/*&tag=* Disallow:/*?month=* Disallow:/*&month=* Disallow:/*?view=* Disallow:/*&view=* Disallow:/*?format=json Disallow:/*&format=json Disallow:/*?format=page-context Disallow:/*&format=page-context Disallow:/*?format=main-content Disallow:/*&format=main-content Disallow:/*?format=json-pretty Disallow:/*&format=json-pretty Disallow:/*?format=ical Disallow:/*&format=ical Disallow:/*?reversePaginate=* Disallow:/*&reversePaginate=* Any ideas?
Getting Started | | andrewrench0 -
Questions about the MOZ
Sometimes the MOZ logs out of my account on its own, and I'm quite annoyed about this, how can it not log out of the website even though I've already saved it?
Product Support | | gogoanimetp0 -
Why doesn't Moz read the right meta description?
We have a WordPress blog and Moz continues to read our site title as the meta description instead of the nicely crafted and SEO-friendly meta descriptions we are writing on each post in our Yoast plugin. Can someone help use understand why Moz is reading the site title rather than the post meta descriptions? (We did test deleting the site title all together but then this made Moz think that there is no meta description for any of the posts.)
Product Support | | simran-s1 -
Crawl Issue
Hi, We have 3 campaigns running for our websites in different territories. All was going well until April 11th when Moz reported that our .com site (sendmode.com) could not be crawled. I get this error "Your page redirects or links to a page that is outside of the scope of your campaign settings ..." I've been through the site a number of times but have been unable to get to the root of the problem. Robots.txt and 301's look fine. Is there any way I can find out which page is causing the issue? John
Product Support | | johnmc330 -
Moz cant crawl site?
We're getting an error saying Moz is getting an errors crawling our client's site, but when I've put this though Google Search Console I'm not seeing any issues - any suggestions?
Product Support | | Ramarketingrob0 -
Emails from Moz display incorrect report information
Hey, I have noticed that for the last 3 weeks or so the emails from Moz have been displaying incorrect information. The results it shows in the email seem to be from previous crawls. For example The previous week a sitemap that didn't update that caused 400+ pages to appear as 404's. This wasn't reported in the email and it stated there were only 2 errors on the site. (Of which we knew about anyway and choose to ignore). Thankfully i checked the crawl report anyway out of habit and saw the onsite report to show 400+ crawl errors. I went on to fix these errors. This week, the crawl report showed 400+ crawl errors, I immediately logged on to the website to check the campaign to find there are only the 2 expected crawl errors. It is almost as if the info from the campaign is being pulled and emailed to me seconds before it is updated from the crawl info. I am finding this to be the case in both the keyword and crawl errors email reports.
Product Support | | ATP0 -
SEO Campaign for testing site?
We are in the process of a website redesign and have initiated an SEO campaign on the current site to set benchmarks. Does it make sense for us to run a separate SEO campaign on the testing site as well? Michelle
Product Support | | Michelle-Meyer0 -
MOZ Crawl help
Our MOZ report says it crawled 1800 pages so it reports a lot of errors based on those pages. We don't have that many pages on our site. What is MOZ crawling? I updated the profile to make sure it crawls the filtered page section of Google Analytics.
Product Support | | JessiK0