Is this Duplicate Content?
-
I searched a snippet of one of our Articles (in quotes) and got two results back in Google, one for the article on our site and one for our development/staging site. Does that mean that our development site is getting indexed by Google, even thought we "Disallow:/" in the robots.txt file? Is this a big duplicate content issue?
Thanks
-
I personally have had less of a problem with robots.txt not being followed by Google than I have had developers muck with the robots.txt file. In my particular case, they would update the development site with information from the live site (as we were adding more content to a database on the live site and they were syncing it). They'd copy over the live robots.txt as well, and suddenly the development site was being indexed.
I use Code Monitor for all of my robots.txt files, live or development, so I can be notified when they change. https://www.polepositionweb.com/roi/codemonitor/
I suggest verifying the development site in Google Webmaster Tools. From there, you can request the content to be removed (after you've blocked in robots.txt or used the meta noindex), and you can also use their robots.txt tool to see if there was possibly an error in the file.
-
Yes it does, before no-indexing the page with meta tag, you should put a rel anoical to the right page, as I find the duplicate can hang arounbd in the index for some time. I would let it be crawled again so that the canonical tag can be found before you remove it
-
Yes, you are actually showing 2 shadow copies of your website to Google.
Use a nofollow, noindex tag in the meta. Also, it would be a good idea to have a rel=canonical in place for your testwebsite.(just in case).
-
Yes, that is duplicate content and is an issue. Robots.txt is unreliable at best and there are very stringent guidelines to using it properly.
Instead, use an actual meta tag of noindex! That will solve the issue.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
I want to use some content that I sent out in a newsletter and post as a blog, but will this count as duplicate content?
I want to use some content that I sent out in a newsletter a while ago - adding it as a blog to my website. The newsletter exists on a http://myemail.constantcontact.com URL and is being indexed by Google. Will this count as duplicate content?
Content Development | | Wagada0 -
When Somebody Copies your content What should you do?
I wrote an article in 2011 A Brief History of Benjamin Moore Paint on my website ( I am a painting contractor). It is a content creators win; sited as an authoritative article on wikipedia. Benjamin Moore Paint company has copied my content on their own corporate website; a deep page. Is this hurting, helping, or neutral? bX1kFQg,639HqZE bX1kFQg,639HqZE#1
Content Development | | johnshearer0 -
Duplicates from weird domains
My sit is http://www.webdesign.org/, but the other day I found these sites that have duplicates of my original site: <a>http://wordpresswww.webdesign.org/</a><a>http://fdsfsdswww.webdesign.org/</a><a>http://gfdgdfgdfgwordpresswww.webdesign.org/</a><a>http://w54354353w.webdesign.org/</a>http://wojhkhjkhw.webdesign.org/What really freaks me out is that the content on those sites is 100% up to date. Same as on http://www.webdesign.org/. Now here's my question. 1. Since my site http://www.webdesign.org/ is the root domain, I take it that I can somehow disable those sites (subdomains) from my domain 'admin panel' or what not?2. If you can use those subdomains even if you don't own the root domain (http://www.webdesign.org/), it looks that some negative SEO has been done to my site?Which of my assumptions are right? Please help me to figure that out.
Content Development | | VinceWicks0 -
What makes high quality content?
Content is becoming more and more important in rankings and I was wondering what exactly Google defines a good content. Any ideas?
Content Development | | EJDekkers0 -
Content Architecture - Breakout Pages
If you have a page that summarizes four different product types adequately in a chart that requires no scroll, is there an SEO justification to also breaking out each product into a separate page, but basically it would contain the same information? The SEO in me says yes, because that's more crawlable content you can optimize, but wouldn't it go against usability and general common sense?
Content Development | | SSFCU0 -
What are the best content writer sites?
Hi, I'm doing some work on a new blog and wondered if anyone could recommend some low cost content writers? I have only justed started researching this service, so any advice the SEOmoz community could give would be grately appreciated. Thanks in advance.
Content Development | | RBH0 -
How often should content be updated
With all of Google's recent algo updates (or ranking updates, whatever they're calling it now), we've obviously been looking into changing our content strategy and shifting it from quantity to quality. How often would you say is ideal for website content updates? i.e. should we be updating once a month? Once every couple of months? This isn't a blog - just a regular services-oriented site. My take on it is that it should be as often as organically possible - and that means something different for everyone. At the same time, we want Google coming back frequently to crawl the site. Thanks!
Content Development | | eyecarepro0