Unsolved Moz crawler not crawling on my site
-
Hi all,
im facing an issue where moz crawler is unable to crawl my site. The following error keeps showing Our crawler was banned by a page on your site, either through your robots.txt, the X-Robots-Tag HTTP header, or the meta robots tag.
This is my robots.txt file : https://www.wearefutureheads.com/robots.txt
I'm not sure what else am I missing.. can anyone help
-
@teikh Hi, thanks for contacting us! Would you be able to write into help@moz.com so that our support team can take a closer look at your query please? Thanks
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved What would the exact text be for robots.txt to stop Moz crawling a subdomain?
I need Moz to stop crawling a subdomain of my site, and am just checking what the exact text should be in the file to do this. I assume it would be: User-agent: Moz
Getting Started | | Simon-Plan
Disallow: / But just checking so I can tell the agency who will apply it, to avoid paying for their time with the incorrect text! Many thanks.0 -
Unsolved Strange "?offset" URL found with content crawl issues
I recently recieved a slew of content crawl issues via Moz for URL's that I have never seen before For example:
Moz Pro | | HannahPalamara
Standard URL: https://skilldirector.com/news,
Newly identified URL: https://skilldirector.com/news?offset=1469542207800&category=Competency+Management). Does anyone know where the URL comes from and how to fix it?0 -
How get rid of 403 crawl error?
My wordpress website has 162 crawl 403 errors. Based on what I read it means that the server is denying crawlers to access the pages. The pages itself will load so guessing it's just an issue with crawlers only. How do I go about fixing this issue?
On-Page Optimization | | emrekeserr30 -
Unsolved How do I cancel this crawl?
The latest crawl on my site was the 4th Jan with a current crawl 'in progress'. How do i cancel this crawl and start a new one? I've been getting keyword ranking etc but no new issues are coming through. Screenshot 2022-05-31 083642.jpg
Moz Tools | | ClaireU0 -
How often are keywords updating in moz
How often do they update, I have set up a campaign, added kw and its been 7 day and no update on rankings..
Product Support | | dave8580 -
Emails from Moz display incorrect report information
Hey, I have noticed that for the last 3 weeks or so the emails from Moz have been displaying incorrect information. The results it shows in the email seem to be from previous crawls. For example The previous week a sitemap that didn't update that caused 400+ pages to appear as 404's. This wasn't reported in the email and it stated there were only 2 errors on the site. (Of which we knew about anyway and choose to ignore). Thankfully i checked the crawl report anyway out of habit and saw the onsite report to show 400+ crawl errors. I went on to fix these errors. This week, the crawl report showed 400+ crawl errors, I immediately logged on to the website to check the campaign to find there are only the 2 expected crawl errors. It is almost as if the info from the campaign is being pulled and emailed to me seconds before it is updated from the crawl info. I am finding this to be the case in both the keyword and crawl errors email reports.
Product Support | | ATP0 -
Number of pages crawled = 1; Why?
Since November, we've been trying to figure out why, when I select Crawl Diagnostics, my number of pages crawled is only 1. In mid-november, we changed our URL. That is, we went from www.example.com/home-page/ to www.example.com/new-home-page/. My first assumption was that I needed to re-create my Moz profile. That didn't fix it. The only crawl error we get is the no rel="cannonical" found -- but it's there. We find it on every page, including the home page. Our content shows up in search. Moz bar shows us info for every page. I just don't know what else to check. Everything else in my dashboard seems to look as expected. Specifically, I've turned to Crawl Diagnostics to find 4XX errors on our site. Typically we find one or two per week. Sometimes 0. Sometimes 4 or more. But it's been 0 since November. I highly doubt we've arrived at perfection. Any thoughts?
Product Support | | seo-nicole0 -
Custom Anchor text link on Moz profile section
Hi, Does anyone know if you can modify your anchor text and URL after you reach 200 Moz points on your profile section (On the web section under image)? I am close to getting my 200 pts (btw...enjoyed the entire journey) and am trying to choose my anchor text. Here is some interesting analysis too. I have a Spanish site since we are an Argentinean firm. 1. Would a spanish word as anchor text have any value since Moz.com is an English site? I am think about semantics here and how Google likes only content relevant links. My thought is that it does today but it may not be true in the future if Google is going to do away with linkbuilding and just do semantics and co-citation. Thanks Carla
Product Support | | Carla_Dawson0