"Issue: Duplicate Page Content " in Crawl Diagnostics - but these pages are noindex
-
Saw an issue back in 2011 about this and I'm experiencing the same issue. http://moz.com/community/q/issue-duplicate-page-content-in-crawl-diagnostics-but-these-pages-are-noindex
We have pages that are meta-tagged as no-everything for bots but are being reported as duplicate. Any suggestions on how to exclude them from the Moz bot?
-
Technically that could be done in your robots.txt file but I wouldn't recommend that if you want Google to crawl them too. I'm not sure if Rogerbot can do that. Sorry I couldn't be more help.
If you don't get one of the staffers on here in the next few days, I would send a ticket to them for clarification.
If you decide to go with robots.txt here is a resource from Google on implementing and testing it. https://support.google.com/webmasters/answer/156449?hl=en
-
Thanks for the information on Rogerbot. I understand the difference between the bots from Google and Moz.
Some errors reported in Moz are not real. For example we use a responsive slider on the home page that generates the slides from specific pages. These pages are tagged to no-everything so as to be invisible to bots, yet they are generating errors in the reports.
Is there anyway to exclude some pages from the reports?
-
Don't forget that Rogerbot (moz's crawler) is a robot and not an index like Google. Google used robots to gather the data but the results we see is an index. Rogerbot will crawl the pages regardless of noindex or nofollow.
Here is more info on RogerBot http://moz.com/help/pro/rogerbot-crawler
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Sale page ranking for "[blank] for sale" keywords
My company's Ecommerce site has a sale category that is currently out performing some of our normal categories in the SERPs for "[blank] for sale" keywords. For example the sale category landing page is ranking for the keyword "vegetable seeds for sale" rather than the vegetable seed category. Has anyone ever dealt with a similar situation to this? or does anyone have general advice for optimizing (or weakening) sale pages?
On-Page Optimization | | Scoleman1 -
Duplicate Page Content - default.html
I am showing a duplicate content error in moz. I have site.com and site.com/default.html How can I fix that? Should I use a canonical tag? If so, how would i do that?
On-Page Optimization | | bhsiao0 -
What constitutes duplicate content on a page?
I am working on SEO for a Shopify store. Their products are very similar, hence the pages are so similar that Moz shows them as duplicate content. The only difference in the product pages is the title and model number. I am going to "go for the gold" and try re-writing all the product descriptions. It's incredibly difficult due to the products being nearly identical with just a minor variation. I know I could go down the road of just creating variants --- but the customer is not down for that. Here's my question: what constitutes duplicate content? 80% of the content, 90%???? If I can going to re-write the descriptions, what should I aim for? Thank you!
On-Page Optimization | | steve_linn1 -
I am trying to better understand solving the duplicate content issues highlighted in your recent crawl report of our site - www.thehomesites.com.
Below are some of the urls highlighted as having duplicate content -
On-Page Optimization | | urahul
http://www.thehomesites.com/zip_details/76105
http://www.thehomesites.com/zip_details/44135
http://www.thehomesites.com/zip_details/75227
http://www.thehomesites.com/zip_details/94501 These are neighborhood reports generated for 4 different zip codes. We use a standard template to create these reports. What are some of the steps we can take to avoid these pages being categorized as duplicate content?0 -
Duplicate content, which seems not to be duplicate :S
After crawling I am used to getting a lot of duplicate content messages in Moz, which are High Priority. I do not know what to do with them, since I believe we tackled all the issues. Main point being the advise to put in a link rel=canonical. An example of a page that accordeing to the report has a duplicate. I do not see how. Can you help with that? http://www.beat-it.nl/4y6hctr24x7wdmr-ml350-p-ic-procaresvc.html duplicate sample http://www.beat-it.nl/modu-hp-a5800-acm-for-64-256-aps.html
On-Page Optimization | | Raymo0 -
"og:description" vs. name="description"
According to Rock Your SEO with Structured Social Sharing "OG description overrides meta description tag." Moz Crawl Diagnostics seems to ignore og:description and only look for meta name="description" - does that mean my meta descriptions tags should be meta name?
On-Page Optimization | | leighw0 -
Internal Duplicate Content/Canonical Issue/ or nothing to worry about
Unfortunately, my developer cannot give me an answer to this so I really do hope someone can help. The homepage of my website is http://www.laddersfree.co.uk however I also have a page http://www.laddersfree.co.uk/index.php that has a page rank and essentially duplicates the home page. Does someone know what this is? Do I need to get my developer to do a 404? It is worrying that he has not come back to me. Thanks Jason
On-Page Optimization | | gymmad0 -
Duplicate Page Titles and Duplicate Content
I've been a Pro Member for nearly a year and I am bound and determined to finally clean up all the crawl errors on our site PracticeRange.com. We have 180 errors for Duplicate Page Titles and Duplicate Content. I fixed many of the pages that were product pages with duplicate content. Those product descriptions were edited and now have unique content. However, there remain plenty of the errors that are puzzling. Many of the errors reference the same pages, for example, the Home Page, Login Page and the Search page (our catalog pages).
On-Page Optimization | | AlanWills
In the case of the Catalog Page errors, these type pages would have the same title every time "Search" and the results differ according to category. http://www.practicerange.com/Search.aspx?m=6
http://www.practicerange.com/Search.aspx?m=15 If this is rel=canonical issue, how do I fix it on a search result page? I want each of the different category type pages to be indexed. One of them is no more important than the other. So how would I incorporate the rel=canonical? In the case of the Home Page errors, I'm really confused. I don't know where to start to fix these. They are the result of a 404 error that leads to the home page. Is the content of the 404 page the culprit since it contains a link to the home page? Here are examples of the Home Page type of crawl errors. http://www.practicerange.com/404.aspx?aspxerrorpath=/Golf-Training-Aids/Golf-Nets/~/Assets/ProductImages/products/Golf-Training-Aids/Rubber-Wooden-Tee-Holder.aspx http://www.practicerange.com/404.aspx?aspxerrorpath=/Golf-Training-Aids/Golf-Nets/~/Assets/ProductImages/products/Golf-Training-Aid/Impact-Bag.aspx Thanks , Alan WillsPracticeRange.com0
