"Issue: Duplicate Page Content " in Crawl Diagnostics - but these pages are noindex
-
Saw an issue back in 2011 about this and I'm experiencing the same issue. http://moz.com/community/q/issue-duplicate-page-content-in-crawl-diagnostics-but-these-pages-are-noindex
We have pages that are meta-tagged as no-everything for bots but are being reported as duplicate. Any suggestions on how to exclude them from the Moz bot?
-
Technically that could be done in your robots.txt file but I wouldn't recommend that if you want Google to crawl them too. I'm not sure if Rogerbot can do that. Sorry I couldn't be more help.
If you don't get one of the staffers on here in the next few days, I would send a ticket to them for clarification.
If you decide to go with robots.txt here is a resource from Google on implementing and testing it. https://support.google.com/webmasters/answer/156449?hl=en
-
Thanks for the information on Rogerbot. I understand the difference between the bots from Google and Moz.
Some errors reported in Moz are not real. For example we use a responsive slider on the home page that generates the slides from specific pages. These pages are tagged to no-everything so as to be invisible to bots, yet they are generating errors in the reports.
Is there anyway to exclude some pages from the reports?
-
Don't forget that Rogerbot (moz's crawler) is a robot and not an index like Google. Google used robots to gather the data but the results we see is an index. Rogerbot will crawl the pages regardless of noindex or nofollow.
Here is more info on RogerBot http://moz.com/help/pro/rogerbot-crawler
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Consolidating a Large Site with Duplicate Content
I will be restructuring a large website for an OEM. They provide products & services for multiple industries, and the product/service offering is identical across all industries. I was looking at the site structure and ran a crawl test, and learned they have a LOT of duplicate content out there because of the way they set up their website. They have a page in the navigation for “solution”, aka what industry you are in. Once that is selected, you are taken to a landing page, and from there, given many options to explore products, read blogs, learn about the business, and contact them. The main navigation is removed. The URL structure is set up with folders, so no matter what you select after you go to your industry, the URL will be “domain.com/industry/next-page”. The product offerings, blogs available, and contact us pages do not vary by industry, so the content that can be found on “domain.com/industry-1/product-1” is identical to the content found on “domain.com/industry-2/product-1” and so-on and so-forth. This is a large site with a fair amount of traffic because it’s a pretty substantial OEM. Most of their content, however, is competing with itself because most of the pages on their website have duplicate content. I won’t begin my work until I can dive in to their GA and have more in-depth conversations with them about what kind of activity they’re tracking and why they set up the website this way. However, I don’t know how strategic they were in this set up and I don’t think they were aware that they had duplicate content. My first thought would be to work towards consolidating the way their site is set up, so we don’t spread the link-equity of “product-1” content, and direct all industries to one page, and track conversion paths a different way. However, I’ve never dealt with a site structure of this magnitude and don’t want to risk messing up their domain authority, missing redirect or URL mapping opportunities, or ruin the fact that their site is still performing well, even though multiple pages have the same content (most of which have high page authority and search visibility). I was curious if anyone has dealt with this before and if they have any recommendations for tackling something like this?
On-Page Optimization | | cassy_rich0 -
How to optimize WordPress Pages with Duplicate Page Content?
I found the non WWW ans WWW duplicate pages URL only, more than thousand pages.
On-Page Optimization | | eigital0 -
Rel="canonical" at the same page
Hello Everyone!! We have a Joomla Site and in the template we have a php function that create the **link rel="canonical" **and in the href inserts the same page url. For example, if the we do a search and the url have some cookies. That Url is gonna be the **rel="canonical" **for that page. Is it working correctly? We need an advice to to set it up correctly! Thanks!
On-Page Optimization | | mycostaricalink0 -
Google is showing the "wrong" page in the serps for user queries
Our site, cpuboss.com has two main types of pages: comparisons of CPUs, and pages about individual CPUs. When I google search for "fx8350", a keyword for the cpu "AMD FX-8350", I hoped to see our CPU page ranking, instead I see a comparison of this CPU against another. http://i.imgur.com/QwNczDj.png?1 (Our page ranks #5) Any idea? Many of our comparisons rank for CPU keywords, when I think it'd be a better experience for the user if our cpu pages ranked instead. This particular CPU page is linked to on our homepage with appropriate anchor text, and linked to on that comparison also. I would have hoped the relevance of the cpu page to this keyword would be stronger than the comparison page, and google might use that relevance signal to show the user the right page. thoughts?
On-Page Optimization | | cdnventure0 -
Duplicate Content Again
Hello Good People. I know that this is another duplicate post about duplicate content (boring) but i am going crazy with this.. SeoMoz crawl and other tools tells me that i have a duplicate content between site root and index.html. The site is www.sisic-product.com i am going crazy with this... the server is IIS so cannot use htaccess please help... thanks
On-Page Optimization | | Makumbala0 -
How are your "Service Area" pages handling Penguin/Panda?
We just got a new client because of recent Penguin/Panda changes. A national "SEO" firm decided it was a good idea to set up a page for each service town or county they serve with nothing but duplicate content. Needless to say, on the week of the 23rd, their rankings tanked from 1st page (it's not a competitive niche) to 4th. I'm not bringing this up to brag, but rather because it got me thinking... How are your geographically targeted "service area" pages doing? Have the recent changes caused you to rethink your geographic targeting in any way?
On-Page Optimization | | BedeFahey0 -
Crawl Diagnostics - Duplicate Content and Duplicate Page Title Errors
I am getting a lot of duplicate content and duplicate page title errors from my crawl analysis. I using volusion and it looks like the photo gallery is causing the duplicate content errors. both are sitting at 231, this shows I have done something wrong... Example URL: Duplicate Page Content http://www.racquetsource.com/PhotoGallery.asp?ProductCode=001.KA601 Duplicate Page Title http://www.racquetsource.com/PhotoGallery.asp?ProductCode=001.KA601 Would anyone know how to properly disallow this? Would this be as simple as a robots.txt entry or something a little more involved within volusion? Any help is appreicated. Cheers Geoff B. (a.k.a) newbie.
On-Page Optimization | | GeoffBatterham0 -
Will duplicate content supplied from a hotel provider damage my website, or simply just the pages that it appears on?
Hi, I currently have a lot of hotel listings pages with little or no content, as I'm scared that if I place duplicate hotel descriptions on the pages then Google will stop ranking the page. I've found that having descriptions of some kind do help conversion significantly, so I'm considering generating unique hotel descriptions on each main page (page 1 in each set of listings) - these are the pages that Google indexes. On subsequent pages (page 2, page 3 etc.) I'm thinking about resorting to displaying the duplicate affiliate content hotel descriptions - these pages can be crawled but are set to noindex. My question is, do you think this is likely to have an effect on my website in the rankings, and as a result push my primary pages (that contain 100% unique content) down in SERPs. Thanks Mike
On-Page Optimization | | mjk260