Is this Duplicate Content?
-
I searched a snippet of one of our Articles (in quotes) and got two results back in Google, one for the article on our site and one for our development/staging site. Does that mean that our development site is getting indexed by Google, even thought we "Disallow:/" in the robots.txt file? Is this a big duplicate content issue?
Thanks
-
I personally have had less of a problem with robots.txt not being followed by Google than I have had developers muck with the robots.txt file. In my particular case, they would update the development site with information from the live site (as we were adding more content to a database on the live site and they were syncing it). They'd copy over the live robots.txt as well, and suddenly the development site was being indexed.
I use Code Monitor for all of my robots.txt files, live or development, so I can be notified when they change. https://www.polepositionweb.com/roi/codemonitor/
I suggest verifying the development site in Google Webmaster Tools. From there, you can request the content to be removed (after you've blocked in robots.txt or used the meta noindex), and you can also use their robots.txt tool to see if there was possibly an error in the file.
-
Yes it does, before no-indexing the page with meta tag, you should put a rel anoical to the right page, as I find the duplicate can hang arounbd in the index for some time. I would let it be crawled again so that the canonical tag can be found before you remove it
-
Yes, you are actually showing 2 shadow copies of your website to Google.
Use a nofollow, noindex tag in the meta. Also, it would be a good idea to have a rel=canonical in place for your testwebsite.(just in case).
-
Yes, that is duplicate content and is an issue. Robots.txt is unreliable at best and there are very stringent guidelines to using it properly.
Instead, use an actual meta tag of noindex! That will solve the issue.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Is there a good tool for finding Duplicate Content?
Hi! Are there any recommendation voor Duplicate Content finder tools? Is it offered on MOZ (not the one that belongs to 'seo issues') or a different site?
Content Development | | Marketing-SurpriseFactory1 -
Content Duplication Issue On Content Publishing Site
I am running Free Article Posting site but I discover people are posting their content which already has been published on different sites before. What should i need to do in order to save my site from Google penalty.3 waiting for your kind help in this regard Thanks in advance
Content Development | | Mustansar0 -
Content Spinner Tool??? Worth? Recommendations?
Hi all !! We have different websites in the english language (UK, IE, US, etc...), but we focus our content strategy (landing pages) in just one of them. We were thinking about using a Content spin tool to use this content in the other websites and give them an extra push in terms of SEO (content is the king :)). Did you have any idea if they work fine? Did you have any experience with them? Can you recommend any? Are they tools really worth or they are not working fine and after the spin a full review (and probably some re-write) is necessary? Of course I´m talking about the paid ones.... Not even thinking about the free tools 😉 Thanks for your help in advance !!!
Content Development | | AutoEurope0 -
Are Duplicate Bio's Duplicate Content?
I'm wondering if I need to go through all the various bio's our firm has on all the various legal directory and client review sites to make sure they have unique bio's? I really, really don't want to do that, but if they are going to flagged as duplicate content, I will. I'm hoping sites like "findlaw," "avvo," etc, have some built in rel:cannonical or something that says these bio's aren't to be seen as unique content and therefore conflict with what's on our site. Anybody know? Just to clarify, any of the sties that have asked for unique content/bio/about us, I have complied with that. However, a lot don't specifically state it has to be unique, so I've just copy and pasted from our site in those cases. Thanks, Ruben
Content Development | | KempRugeLawGroup0 -
Duplicates from weird domains
My sit is http://www.webdesign.org/, but the other day I found these sites that have duplicates of my original site: <a>http://wordpresswww.webdesign.org/</a><a>http://fdsfsdswww.webdesign.org/</a><a>http://gfdgdfgdfgwordpresswww.webdesign.org/</a><a>http://w54354353w.webdesign.org/</a>http://wojhkhjkhw.webdesign.org/What really freaks me out is that the content on those sites is 100% up to date. Same as on http://www.webdesign.org/. Now here's my question. 1. Since my site http://www.webdesign.org/ is the root domain, I take it that I can somehow disable those sites (subdomains) from my domain 'admin panel' or what not?2. If you can use those subdomains even if you don't own the root domain (http://www.webdesign.org/), it looks that some negative SEO has been done to my site?Which of my assumptions are right? Please help me to figure that out.
Content Development | | VinceWicks0 -
Content
Hi, We produce research report on same topic every year (e.g. cost of patrol in different part of the country) and publish them to a new url every time. What is the best way to do it? create new page every year or update the existing research report with modification Considering long term benefits. Thanks
Content Development | | APH0 -
Question - Hidden Content via 'Read More;'
Hi, Does having text in hidden areas (that are expandable with a button) harm their value in SEO? E.g. there are 'read more' areas on the homepage of this site that have content that is not accessible otherwise. Does this content have the same weight when it is hidden versus if this whole area was visible? http://goo.gl/QQvAZ
Content Development | | qlkasdjfw0 -
Is it advisable to have unique pages for different cities/states though there wouldnt be any actual differentiation in the actual content.
Is it advisable to have unique pages for different cities/states though there wouldnt be any actual differentiation in the content. For example should we have separate pages for "hammers in california" & "hammers in new york". The product is same and content more or less the same. The search volume for individual queries is low but collectively makes a large number. The unique title tag automatically will generate traffic. So does it make sense to make 50 such pages. Else is there any way to uniquely target 50 such queries/month/city
Content Development | | DYo0