Standard Syntax in robots.txt doesn't prevent Moz bot from crawling
-
A client is getting many false positive site crawl errors for things like duplicate titles and duplicate content on pages that include /tag/ in the URL. An example is https://needquest.com/place_tag/autism-spectrum-disorder/page/4/
To resolve this we have set up a disallow statement in the robots.txt file that says
Disallow: /page/For some reason this appears not to work, as the site crawl errors continue to list pages like this. Does anyone understand why that would be and what we need to do to properly disallow crawling these pages?
-
Thanks, Tawny,
If you look at Duplicate titles, check the first one (https://needquest.com/place_tag/autism-spectrum-disorder/). All the URLs with a duplicate title have /page/ in them. I will suggest they move the Allow statement and see if that helps.
-
I'm not seeing that URL coming up with Duplicate Title or Duplicate Content issues — when I search by that URL I see no Content issues at that URL. I do see that URL in the All Crawled Pages section, but I can't find it bringing up Content issues in the app.
That said, I took a look at your robots.txt file, and I think this could be a result of having an Allow command before the rest of the Disallow commands. I think possibly if you put that Allow command at the end of the block of Disallow commands, rogerbot would see the disallow for /page/ and stop crawling those URLs.
If you're still running into trouble, I would suggest writing in to us at help@moz.com so we can take a closer look at the Campaign and what could be going on there.
-
Any reason the Disallow: /page/ isn't preventing URLs like
https://needquest.com/place_tag/autism-spectrum-disorder**/page/**4/
from generating duplicate descriptions and title errors in our site crawl? It was my hope that those pages wouldn't be crawled at all. -
Sorry, Tawny ... I did go back and correct y question. We did apply Disallow: /page/ to address this issue. The /place_tag/ is found in many pages we DO want to crawl and index ... and we only want here to disallow those page 2, page 3, page 4, etc. pages.
(We also disallowed /tag/, /category/, and a few other common issues that generate false positives in the site crawl.)
-
Hey there!
Tawny from Moz's Help Team here.
Adding a disallow directive for /tag/ won't help with the example URL you've provided — that URL doesn't have /tag/ in the URL pathway. To block us from seeing content like that URL you listed, you'd need a disallow directive for /place_tag/.
If you include that disallow directive, that should stop us from seeing duplicate content on pages with /place_tag/ in the URL.
Hope that helps! If you've still got questions, feel free to shoot us a note over at help@moz.com and we'll do our best to sort things out with you.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved What are the top 5 things you recommend you should do when you first subscribe to Moz?
Hi
Getting Started | | djnlowe
Im new to Moz and would like to get the most out of it as quickly as possible. so like the subject say - What are the top 5 things you would recommend I should to get the most out of Moz? #getstarted0 -
I used moz tool and it was very useful to me.
I used moz tool for 3 months. Before that my website never ranked. But now it started ranking. Thanks to moz. My website is https://www.designcafe.com
Getting Started | | designcafe1 -
What happened to Moz perks?
Not sure if it completely disappeared or just moved somewhere else. Would be curious to hear!
Getting Started | | Joe.Robison2 -
Moz Pro for Local Catering Company?
Is Moz Pro or Moz Content useful for a local catering company? I've started a trial to try to determine if it will benefit us. Our new site will be up next week so we don't currently have traffic. We will be using some Adsense and direct mail campaigns to get started, but not sure if Moz local will be sufficient. Please help connecting the dots. I know it would be good to rank higher for example "Catering Tahoe, Ca." But how would Moz help me see what kind of content to create for a local catering company? I'm not sure a blog article makes sense.
Getting Started | | Nasoj0 -
Rel Canonical Notice in Moz Analytics
Hello all. I just received my first report fro Moz Analytics. Signed up here to see what problems my site might have and find out how to fix them. On all of my issue pages I see notice with black dot saying that my urls are Rel Canonical Using rel=canonical suggests to search engines which URL should be seen as canonical. I know that recently my web developer set up so that every url on my website has a rel=canonical as I think there was an issue with trailing slash. Is this to be considered an issue or I can just leave it? Thank you guys. Regards, Armands
Getting Started | | A_Fotografy0 -
Oops! This doesn't appear to be a valid URL. Please try again.
Hi everyone! Looking for a bit of help here. This message keeps popping up when I try to put in my url (www.qltyctrl.com). I've tried qltyctrl.com, www.qltyctrl.com, and http://www.qltyctrl.com and nothing works. Can someone help me out here?
Getting Started | | Nimble.Keith0 -
I can't add Google+
I just signed up for the beta analytics, when trying to add our CEO Google plus it asks to create a new G+ profile? Why can't I add her Google + without creating a new one?
Getting Started | | KatherineKotaw0