"Issue: Duplicate Page Content " in Crawl Diagnostics - but these pages are noindex
-
Saw an issue back in 2011 about this and I'm experiencing the same issue. http://moz.com/community/q/issue-duplicate-page-content-in-crawl-diagnostics-but-these-pages-are-noindex
We have pages that are meta-tagged as no-everything for bots but are being reported as duplicate. Any suggestions on how to exclude them from the Moz bot?
-
Technically that could be done in your robots.txt file but I wouldn't recommend that if you want Google to crawl them too. I'm not sure if Rogerbot can do that. Sorry I couldn't be more help.
If you don't get one of the staffers on here in the next few days, I would send a ticket to them for clarification.
If you decide to go with robots.txt here is a resource from Google on implementing and testing it. https://support.google.com/webmasters/answer/156449?hl=en
-
Thanks for the information on Rogerbot. I understand the difference between the bots from Google and Moz.
Some errors reported in Moz are not real. For example we use a responsive slider on the home page that generates the slides from specific pages. These pages are tagged to no-everything so as to be invisible to bots, yet they are generating errors in the reports.
Is there anyway to exclude some pages from the reports?
-
Don't forget that Rogerbot (moz's crawler) is a robot and not an index like Google. Google used robots to gather the data but the results we see is an index. Rogerbot will crawl the pages regardless of noindex or nofollow.
Here is more info on RogerBot http://moz.com/help/pro/rogerbot-crawler
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Content on product category pages - does Google care?
Hi All, I've always been unsure about the importance of content on product category pages. Nobody reads it. If you search for "living room chairs", you're just going to want to see a big list of living room chairs - not read content about living room chairs, how to choose one, etc. On virtually any ecommerce site, category pages have a paragraph or two of total bla-bla. Does this have any impact on search rankings? More specifically, will Googlebot see content on how to choose a living room chair and say "Yes! This is really helpful content"? Or, will it realize that the searcher intent on this keyword is really just to see a list of chairs, and ignore this content - or at least downplay its importance? WDTY?
On-Page Optimization | | BarryBuckman0 -
Identifying Duplicate Page Title
Moz weekly reports, among other things, the "Duplicate Page Title". How can I identify which two urls/pages have duplicate page titles? Is there any simple way to trace?
On-Page Optimization | | Sequelmed0 -
"irrelevant pages of a site"
Hi there! Some pages of my site like "contact" or "registration": Should they have a title and a description tag? They are pages that I don't want them to be shown in the SERPs....Could I be penalized by google If I don't do so? The SEOMOZ crawling tool warms me about this issue (to short titles, no meta-description tags....) Many thanks
On-Page Optimization | | juanmiguelcr0 -
Duplicate Page Content Should we 301 - Best Practices?
What would be the best way to avoid a Duplicate Page Content for these type of pages. Our website generates user friendly urls, for each page..
On-Page Optimization | | 365ToursSafaris
So it is the same exact page, just both versions of the url work.. Example: http://www.safari365.com/about-africa/wildebeest-migration http://www.safari365.com/wildebeest-migration I don't think adding code to the page will work because its the same page for the incorrect and correct versions of the page. I don't think i can use the URL parameter setting because the version with /about-africa/ is the correct (correct as it it follows the site navigation) I was thinking of using the htaccess to redirect to the correct version.. Will that work ? and does it follow best Practices ? any other suggestions that would work better ?0 -
No Content on home page + rankings
If a home page has no content will it hurt the sites ability to rank? The interior pages will have content but not the home page. (See attached image) My client does not want content on the home page as he feels it will take away from the look and feel he wants to achieve. This website is actually 10 sites or locations in one as we intend to market each location (a total of 10) separately. In reality the home page is a doorway page to each separate location. I'd like feedback if possible as to the necessity or not, of content on the Home Page of this or any website. Will the lack of content hurt on the Homer Page hurt with SEO? Thanks
On-Page Optimization | | fun52dig
Gary Downey bobby-vans.jpg0 -
Duplicate Content for Spanish & English Product
Hi There, Our company provides training courses and I am looking to provide the Spanish version of a course that we already provide in English. As it is an e-commerce site, our landing page for the English version gives the full description of the course and all related details. Once the course is purchased, a flash based course launches within a player window and the student begins the course. For the Spanish version of the course, my target customers are English speaking supervisors purchasing the course for their Spanish speaking workers. So the landing page will still be in English (just like the English version of the course) with the same basic description, with the only content differences on that page being the inclusion of the fact that this course is in Spanish and a few details around that. The majority of the content on these two separate landing pages will be exactly the same, as the description for the overall course is the same, just that it's presented in a different language, so it needs to be 2 separate products. My fear is that Google will read this as duplicate content and I will be penalized for it. Is this a possibility or will Google know why I set it up this way and not penalize me? If that is a possibility, how should I go about doing this correctly? Thanks!
On-Page Optimization | | NiallTom0 -
Duplicate Content
Hi I have Duplicate content that i do sent understand 1 - www.example.dk 2- www.example.dk/ I thought i was the same page, whit and without the / Hope someone can help 🙂
On-Page Optimization | | seopeter290 -
Duplicate content Issue
I'm getting a report of duplicate title and content on: http://www.website.com/ http://www.website.com/index.php Of course, they're the same pages but does this need to be corrected somehow. Thanks!
On-Page Optimization | | dbaxa-2613380