"Issue: Duplicate Page Content " in Crawl Diagnostics - but these pages are noindex
-
Saw an issue back in 2011 about this and I'm experiencing the same issue. http://moz.com/community/q/issue-duplicate-page-content-in-crawl-diagnostics-but-these-pages-are-noindex
We have pages that are meta-tagged as no-everything for bots but are being reported as duplicate. Any suggestions on how to exclude them from the Moz bot?
-
Technically that could be done in your robots.txt file but I wouldn't recommend that if you want Google to crawl them too. I'm not sure if Rogerbot can do that. Sorry I couldn't be more help.
If you don't get one of the staffers on here in the next few days, I would send a ticket to them for clarification.
If you decide to go with robots.txt here is a resource from Google on implementing and testing it. https://support.google.com/webmasters/answer/156449?hl=en
-
Thanks for the information on Rogerbot. I understand the difference between the bots from Google and Moz.
Some errors reported in Moz are not real. For example we use a responsive slider on the home page that generates the slides from specific pages. These pages are tagged to no-everything so as to be invisible to bots, yet they are generating errors in the reports.
Is there anyway to exclude some pages from the reports?
-
Don't forget that Rogerbot (moz's crawler) is a robot and not an index like Google. Google used robots to gather the data but the results we see is an index. Rogerbot will crawl the pages regardless of noindex or nofollow.
Here is more info on RogerBot http://moz.com/help/pro/rogerbot-crawler
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Content on product category pages - does Google care?
Hi All, I've always been unsure about the importance of content on product category pages. Nobody reads it. If you search for "living room chairs", you're just going to want to see a big list of living room chairs - not read content about living room chairs, how to choose one, etc. On virtually any ecommerce site, category pages have a paragraph or two of total bla-bla. Does this have any impact on search rankings? More specifically, will Googlebot see content on how to choose a living room chair and say "Yes! This is really helpful content"? Or, will it realize that the searcher intent on this keyword is really just to see a list of chairs, and ignore this content - or at least downplay its importance? WDTY?
On-Page Optimization | | BarryBuckman0 -
Dynamic links & duplicate content
Hi there, I am putting a proposal together for a client whose website has been optimised to include many dynamic links and so there are many pages with duplicate content: only the page title, h1 and URL is different. My client thinks this isn't an issue. What is the current consensus on this? Many thanks in advance.
On-Page Optimization | | lorraine.mcconechy0 -
Products description from third party vendor creating duplicate content issues?
Hi, I am running my client's e-store. The store sells different products from various vendors. Vendors provide us product descriptions. The problem is that these vendors also give these description to display their products on other similar sites and hence creating duplicate content issue. Thanks.
On-Page Optimization | | Kashif-Amin0 -
"Heading 1" vs. "Title" Style for SEO
In Word, you can specify "Heading 1" text which Google presumably treats the same as an HTML tag. Is there any benefit in using the "Title" style? Is it the equivalent of a web page's title?
On-Page Optimization | | BlueLinkERP0 -
Duplicate content with mine other websites
Hi,
On-Page Optimization | | JohnHuynh
I have a primary website www.vietnamvisacorp.com and beside that I also have some secondary websites with have same contents with primary website. This lead to duplicate content errors. What the best solution for this issue? Please help! p/s: I am a webmaster of all websites duplicate content Thank you!0 -
Duplicate content from category pages?
I have an ecommerce store with different categories for my products. Some products do appear in more than one category, is that an issue even if you end up on the same product page/link? Also, I have a "show all products" category, which I believe creates duplicate content too? What is your take on this? What can I do to solve this? Is it even an issue of duplicate content? All answers are very much appreciated!
On-Page Optimization | | danielpett0 -
What is the best way to manage industry required duplicate Important Safety Information (ISI) content on every page of a site?
Hello SEOmozzer! I have recently joined a large pharmaceutical marketing company as our head SEO guru, and I've encountered a duplicate content related issue here that I'd like some help on. Because there is so much red tape in the pharmaceutical industry, there are A LOT of limitations on website content, medication and drug claims, etc. Because of this, it is required to have Important Safety Information (ISI) clearly stated on every page of the client's website (including the homepage). The information is generally pretty lengthy, and in some cases is longer than the non-ISI content on each page. Here is an example: http://www.xifaxan.com/ All content under the ISI header is required on each page. My questions are: How will this duplicated content on each page affect our on-page optimization scores in the eyes of search engines? Is Google seeing this simply as duplicated content on every page, or are they "smart" enough to understand that because it is a drug website, this is industry standard (and required)? Aside from creating more meaty, non-ISI content for the site, are there any other suggestions you have for handling this potentially harmful SEO situation? And in case you were going to suggest it, we cannot simply have an image of the content, as it may not be visible by all internet users. We've already looked into that 😉 Thanks in advance! Dylan
On-Page Optimization | | MedThinkCommunications0 -
Best practice for franchise sites with duplicated content
I know that duplicated content is a touchy subject but I work with multiple franchise groups and each franchisee wants their own site, however, almost all of the sites use the same content. I want to make sure that Google sees each one of these sites as unique sites and does not penalize them for the following issues. All sites are hosted on the same server therefor the same IP address All sites use generally the same content across their product pages (which are very very important pages) *templated content approved by corporate Almost all sites have the same design (A few of the groups we work with have multiple design options) Any suggestions would be greatly appreciated. Thanks Again Aaron
On-Page Optimization | | Shipyard_Agency0
