Standard Syntax in robots.txt doesn't prevent Moz bot from crawling
-
A client is getting many false positive site crawl errors for things like duplicate titles and duplicate content on pages that include /tag/ in the URL. An example is https://needquest.com/place_tag/autism-spectrum-disorder/page/4/
To resolve this we have set up a disallow statement in the robots.txt file that says
Disallow: /page/For some reason this appears not to work, as the site crawl errors continue to list pages like this. Does anyone understand why that would be and what we need to do to properly disallow crawling these pages?
-
Thanks, Tawny,
If you look at Duplicate titles, check the first one (https://needquest.com/place_tag/autism-spectrum-disorder/). All the URLs with a duplicate title have /page/ in them. I will suggest they move the Allow statement and see if that helps.
-
I'm not seeing that URL coming up with Duplicate Title or Duplicate Content issues — when I search by that URL I see no Content issues at that URL. I do see that URL in the All Crawled Pages section, but I can't find it bringing up Content issues in the app.
That said, I took a look at your robots.txt file, and I think this could be a result of having an Allow command before the rest of the Disallow commands. I think possibly if you put that Allow command at the end of the block of Disallow commands, rogerbot would see the disallow for /page/ and stop crawling those URLs.
If you're still running into trouble, I would suggest writing in to us at help@moz.com so we can take a closer look at the Campaign and what could be going on there.
-
Any reason the Disallow: /page/ isn't preventing URLs like
https://needquest.com/place_tag/autism-spectrum-disorder**/page/**4/
from generating duplicate descriptions and title errors in our site crawl? It was my hope that those pages wouldn't be crawled at all. -
Sorry, Tawny ... I did go back and correct y question. We did apply Disallow: /page/ to address this issue. The /place_tag/ is found in many pages we DO want to crawl and index ... and we only want here to disallow those page 2, page 3, page 4, etc. pages.
(We also disallowed /tag/, /category/, and a few other common issues that generate false positives in the site crawl.)
-
Hey there!
Tawny from Moz's Help Team here.
Adding a disallow directive for /tag/ won't help with the example URL you've provided — that URL doesn't have /tag/ in the URL pathway. To block us from seeing content like that URL you listed, you'd need a disallow directive for /place_tag/.
If you include that disallow directive, that should stop us from seeing duplicate content on pages with /place_tag/ in the URL.
Hope that helps! If you've still got questions, feel free to shoot us a note over at help@moz.com and we'll do our best to sort things out with you.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz DA Issue
Hey Community, This is my site which is the Best Drifitng in Australia. I am getting a problem and my site's DA is unable to show using. I want to see it as soon as possible, although I am using a premium tool. Thanks
Getting Started | | jndjue40 -
What does Location National means in Moz Pro Campaigns?
I understand the concept of locations in Moz Pro Campaigns. What does National exactly means? Is it like globally for a United States Google Search Rank or what? Please explain.
Getting Started | | ds9.tech1 -
Clarifications on the Moz Analytics package (Medium - $149 per month)
What are the Moz tools available with this package? What factors of SEO can be checked with these tools? With this package, is it possible to provide a single URL (preferably home page) and Moz will analyse the entire site and highlight how the site performs wrt various SEO factors? This package states that with this package we can run 10 Moz Analytics campaigns. Our understanding of Moz Analytics Campaign is every site; say www.test.com is one analytics campaign. Are we correct? Does the subdomains within a parent domain also considered as one analytics campaign. For e.g., if I have sites: www.mydomain.com and www.xxx.mydomain.com are they considered two separate campaigns or are they considered as one single campaign? In this package it is listed as 750 keywords, what does this signify? In what way this feature can be used to check our site’s SEO compliance. Please elaborate. In this package it is listed as 15 social accounts, what does this signify? In what way this feature can be used to check our site’s SEO compliance. Please elaborate. What do you mean by branded reports?
Getting Started | | WebCCTrial0 -
Changes to Moz
I used to think Moz was rich and full but man, there is so Much new stuff! I love it! I still need to explore everything. Does anybody know if its going to be an option for our clients to log in and see just their account? are changes in the works for reports we can pull and send to clients? thank you!! Matthew
Getting Started | | Mrupp441 -
New to MOZ and working with Web Mentions. Can I use operators?
Our name is HostDime but often put as Host Dime (2 words) by news sources and other sites. How do I set up my brand mention so I only get a notice when both words appear, in order, together. I don't want "That host is a dime" and such. Can I use a +Host +Dime?"Host Dime"? Do these operators work in MOZ?
Getting Started | | hostdime0 -
Getting Redirect Loops in MOZ using Chrome
Been getting bizarre Redirect Loops from Chrome after I log-in to MOZ. Has anyone had something like this happen? I've tried clearing cache, rebooting, etc. but no luck. Thanks in advance!
Getting Started | | danny.wood1 -
I'm new to Moz, where can I find something that explains the basics of everything?
For instance how do I use the Rank Tracker or Keyword Tool. I want to get the most out of Moz and I need a resource that explains it. Thanks
Getting Started | | AliciaMarie1