Unsolved Moz crawler not crawling on my site
-
Hi all,
im facing an issue where moz crawler is unable to crawl my site. The following error keeps showing Our crawler was banned by a page on your site, either through your robots.txt, the X-Robots-Tag HTTP header, or the meta robots tag.
This is my robots.txt file : https://www.wearefutureheads.com/robots.txt
I'm not sure what else am I missing.. can anyone help
-
@teikh Hi, thanks for contacting us! Would you be able to write into help@moz.com so that our support team can take a closer look at your query please? Thanks
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Moz crawler not working
Hi Moz crawler keep failing on my site with the error showing : Our crawler was banned by a page on your site, either through your robots.txt, the X-Robots-Tag HTTP header, or the meta robots tag. I'm not sure what am I missing out.. this is my robots.txt.. i don't think Im missing anything else.. https://www.wearefutureheads.com/robots.txt can the support team help ?
Moz Pro | | teikh0 -
Unsolved What would the exact text be for robots.txt to stop Moz crawling a subdomain?
I need Moz to stop crawling a subdomain of my site, and am just checking what the exact text should be in the file to do this. I assume it would be: User-agent: Moz
Getting Started | | Simon-Plan
Disallow: / But just checking so I can tell the agency who will apply it, to avoid paying for their time with the incorrect text! Many thanks.0 -
Unsolved Strange "?offset" URL found with content crawl issues
I recently recieved a slew of content crawl issues via Moz for URL's that I have never seen before For example:
Moz Pro | | HannahPalamara
Standard URL: https://skilldirector.com/news,
Newly identified URL: https://skilldirector.com/news?offset=1469542207800&category=Competency+Management). Does anyone know where the URL comes from and how to fix it?0 -
How get rid of 403 crawl error?
My wordpress website has 162 crawl 403 errors. Based on what I read it means that the server is denying crawlers to access the pages. The pages itself will load so guessing it's just an issue with crawlers only. How do I go about fixing this issue?
On-Page Optimization | | emrekeserr30 -
On Page Optimization and Page Grader in Moz Needs Some Fixes
Okay, so I've brought this up a couple times to your team and have always been met with resistance and confusion, but I really feel like the fundamental design of your on page grader (the auto one) needs some changes in order to be really useful. Let's start with the basics. An on page grader is supposed to be a tool where you can enter your keyword or keywords and have it scan the URL to see how well the keyword has been implemented on the page. It checks for heading use, anchor text, alt tags, frequency of use, etc. We all should get this if we are here. Now, your manual page grader does just that and does a great job of it. I find it very useful for checking newly created pages, landing pages, competitor pages and the like. However, your auto version found in the Search > On-Page Optimizations is the culprit here. So, according to several staff at Moz, this tool scans your site every week when the rest of your site scan is done. It then takes the rankings of pages for ranked and tracked keywords and assigns a grade for that page. Here's the major problem.... in almost all of my accounts, it very rarely gets the association with keyword and URL correct. Thus, it returns a low page grade even though the grade of the URL would be an A or B for the correct keyword. What seems to be happening is I'm tracking multiple keywords, sometime close variations. This grade tool in a lot of cases is grading a page more than once for several keywords it's ranking for. All the variants it gets a low grade for because the exact keywords isn't in the title, headings, etc. The second thing that seems to be happening is some pages are not being graded using the primary keyword, but instead a variation. In this case it only shows low grades for the page. On one occurrence of trying to discuss this issue with you, I got scoffed at for saying the tool was "guessing at which keyword to use for the grade". Your staff was very adamant in saying "it's not a guess!" My counter point was, call it what you want, it's in accurate to the intended keyword to be optimized for. Now, when one first gets to this tool, all of the above doesn't seem like that big of a deal because there's an "add & manage page grade" section. This section allows you to add and manage page grades. So, one would think that you could go to this tool, add an page grades for keywords that the tool didn't automatically associate and delete ones that it added with an incorrect association and that would be the end of it. However, that's NOT quite the end of it. At your own admission, the tool doesn't function like that. Each week when a new scan is completed, these pages grades essentially reset. So any page grade that you deleted will simply show right back up again. Now, before I go any further, let me explain why this is such an issue for me. First, on page optimization is something my company does and sells as a service and we are pretty good at it. Not saying we're the best, but we're better than most. That said, one of my reasons for choosing Moz was because according to the verbiage on the pages, one can create branded reports to include on page optimization. However, when the tool doesn't get the right keyword to landing page AND there's no way to manually override it, then basically all of my reports make it look like my pages grades suck, when in face they don't. Now, if in my campaigns I only added the exact keyword for each page and didn't do any variations of, then this tools becomes much more accurate. However, I want to track multiple variations. Bottom line is this: I need a page grading and reporting tool I can count on and actually use. Thus far, you solution is riddled with problems and every time I bring it up, you fight me on it saying it's working as it's supposed to. What I'm saying here is "as it's supposed to" is fundamentally wrong. I'd like to discuss this with someone at this point, and preferably someone a bit higher up than a first tier support person because frankly, I'm not getting anywhere. Overall I love your service, but it's things like this that have me questioning wether or not Moz is right. I'm giving you opportunity after opportunity to prove to me that it is. Help me work this out so we can continue that trend. Thanks, Micha
Product Support | | Multiverse-Media-Group4 -
I'm tired of logging in to MOZ
I like this site, I really do, which is why I visit multiple times a day. But I grow weary of these premium sites that require me to log in multiple times a day. This morning I was simultaneously working at ordercup.com, strengthcoach.com, and moz.com, and all three log me out at every turn. Argh!!! What's worse is that some MOZ pages think I'm logged in and others don't. I don't fully understand the logic behind the curtain, so I wander from page to page trying to get the login in sync. Why can't these sites work like lynda.com where I can stay logged in for days and weeks? To top it off, LastPass rolled out a new release that broke auto-login. All this logging in is getting in the way of me getting anything done.
Product Support | | ChristopherGlaeser3 -
I have upgraded my account but I can't add another site. What to do?
Simple, I want to get out of my free trial. I need to work with two/three sites. I seem to have access to five accounts... but I can't add a site. Very frustrating.
Product Support | | SRamsey0