Is this Duplicate Content?
-
I searched a snippet of one of our Articles (in quotes) and got two results back in Google, one for the article on our site and one for our development/staging site. Does that mean that our development site is getting indexed by Google, even thought we "Disallow:/" in the robots.txt file? Is this a big duplicate content issue?
Thanks
-
I personally have had less of a problem with robots.txt not being followed by Google than I have had developers muck with the robots.txt file. In my particular case, they would update the development site with information from the live site (as we were adding more content to a database on the live site and they were syncing it). They'd copy over the live robots.txt as well, and suddenly the development site was being indexed.
I use Code Monitor for all of my robots.txt files, live or development, so I can be notified when they change. https://www.polepositionweb.com/roi/codemonitor/
I suggest verifying the development site in Google Webmaster Tools. From there, you can request the content to be removed (after you've blocked in robots.txt or used the meta noindex), and you can also use their robots.txt tool to see if there was possibly an error in the file.
-
Yes it does, before no-indexing the page with meta tag, you should put a rel anoical to the right page, as I find the duplicate can hang arounbd in the index for some time. I would let it be crawled again so that the canonical tag can be found before you remove it
-
Yes, you are actually showing 2 shadow copies of your website to Google.
Use a nofollow, noindex tag in the meta. Also, it would be a good idea to have a rel=canonical in place for your testwebsite.(just in case).
-
Yes, that is duplicate content and is an issue. Robots.txt is unreliable at best and there are very stringent guidelines to using it properly.
Instead, use an actual meta tag of noindex! That will solve the issue.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Duplicated Terms and Conditions?
Does Google penalise duplicated terms and conditions. I know solicitors use the same legal terms and phrases all the time... Thanks
Content Development | | GaryVictory0 -
Duplicate Content
Hi Does anyone know a site where i can paste text to test for duplication? We've used some outstanding freelancer copywriters in the past but need to check the authenticity of the article created before publishing, Thanks Gary
Content Development | | GaryVictory0 -
Ideas for content
Where do you get your ideas to create content for a blog? I have been using the new keyword planner and am not coming up with any ideas. We are an e-commerce site that deals with commercial equipment. It is hard for me to show anything interesting with products because the company does not own it.
Content Development | | EcommerceSite0 -
Thin/spun content or No content at all?
So I'm working on a site right now that has been plagued with spun/rewritten content..we are talking in the thousands. Now would it be a smarter idea to just remove all the content or leave the content? This is an ecommerce site and I'm tasking a few people to rewrite fresh authentic content. So all of the spun/thin content will be removed eventually. The main question is to just delete it all right now or just replace as we go. I'd like to know what you guys think.
Content Development | | William.Lau0 -
Can I share share my content with other sites either as a individual post or RSS or will I be encourage duplicate content on the web and upset Google
Hi, I have a new site http://www.homeforbusiness.co.uk. I want to encourage traffic to the site by sharing some of my content with other related websites which have a higher PR ranking and traffic for a link to my site. Is this going to upset Google re-duplicate content and devalue my site and stop any organic rankings in the future? Equally some high PR sites which have a good synergy with mine such as http://thewomensbusinessclubs.com/ allow me to add my RSS feed with their blog network. Is this a good thing to do or not for the same reasons as above? Or can I only do the above my creating fresh content? Thanks, Elizabeth Conley
Content Development | | econley0 -
Evergreen content: Dedicated section or blog posts?
As part of our content strategy we are creating an ongoing series of articles to help both our potential buyers and our users learn about our product and improve their knowledge of industry best practices in general. Internally, we've had some debate as to where we should host this content within our site. We've identified two approaches: Series of blog posts Dedicated knowledge section of the website If we go with the first approach, we would created a dedicated section that indexed all the blog posts. If we went with the second, we'd create blog posts for each of the articles announcing their addition. Is there any difference, SEO wise with the two approaches? What would you recommend? Thanks, Darren.
Content Development | | dgibbons0 -
Can you link build without adding any content to the website?
I am an agency-side SEO who has a few different SEO clients. A couple of them absolutely refuse to add any content to their site - no blog, no articles, no link bait, nothing. They have resisted efforts for any content to be placed on their site - whether it is written by us, them, or a third party. They just don't see the value in it, despite my attempts to persuade them. As a result, these websites are just brochure websites. What are your options for link building in this situation? If content is the foundation of white hat link building, what do you do if the client refuses to add content to their site? All help gratefully received! Thanks
Content Development | | kevinmorley0 -
Who gets credit for content
Does it really matter if somebody takes your content changes it slightly and republishes it? This is my original article on history of house paints which was also published on ezine in Feb. and from google alerts I discovered this page http://www.franklinpainting.com/blog/home/a-brief-history-of-house-painting/ a minimally modified version. It is not easy to create content so these folks just copied and made a blog post. Their are now many versions of this on the web..who wins?
Content Development | | johnshearer0