Is this Duplicate Content?
-
I searched a snippet of one of our Articles (in quotes) and got two results back in Google, one for the article on our site and one for our development/staging site. Does that mean that our development site is getting indexed by Google, even thought we "Disallow:/" in the robots.txt file? Is this a big duplicate content issue?
Thanks
-
I personally have had less of a problem with robots.txt not being followed by Google than I have had developers muck with the robots.txt file. In my particular case, they would update the development site with information from the live site (as we were adding more content to a database on the live site and they were syncing it). They'd copy over the live robots.txt as well, and suddenly the development site was being indexed.
I use Code Monitor for all of my robots.txt files, live or development, so I can be notified when they change. https://www.polepositionweb.com/roi/codemonitor/
I suggest verifying the development site in Google Webmaster Tools. From there, you can request the content to be removed (after you've blocked in robots.txt or used the meta noindex), and you can also use their robots.txt tool to see if there was possibly an error in the file.
-
Yes it does, before no-indexing the page with meta tag, you should put a rel anoical to the right page, as I find the duplicate can hang arounbd in the index for some time. I would let it be crawled again so that the canonical tag can be found before you remove it
-
Yes, you are actually showing 2 shadow copies of your website to Google.
Use a nofollow, noindex tag in the meta. Also, it would be a good idea to have a rel=canonical in place for your testwebsite.(just in case).
-
Yes, that is duplicate content and is an issue. Robots.txt is unreliable at best and there are very stringent guidelines to using it properly.
Instead, use an actual meta tag of noindex! That will solve the issue.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Community Discussion - Are data AND storytelling the missing ingredients for successful content marketing efforts?
Data is an important element of content marketing. Storytelling, too, gets readers' attention and has been shown to be instrumental in prospects and customers forming strong connections to brands. But using data and storytelling helps produce some of the strongest content there is to be shared, says Nichole Elizabeth DeMeré in her latest YouMoz article, Here’s How to Combine Storytelling and Data to Produce Persuasive Content. What are your thoughts? Think data and storytelling work best separately? Read the post are share your thoughts below. RS
Content Development | | ronell-smith2 -
Be brutily honest - What do you think of this old content?
My website reports on news relating to certain web hosting providers, and being hit by the latest phantom update a little I am looking back a little more harshly on some of my content with especially the older articles needing alot of work (I know some areas on the site need work), and there are some articles I am just not sure about. I have listed a few OLD articles below https://www.besthostnews.com/the-power-of-shared-web-hosting-by-bluehost/ https://www.besthostnews.com/siteground-sponsors-wordcamp-london/ https://www.besthostnews.com/free-ebook-guide-to-starting-a-website-on-a-budget-a-small-orange/ These are 3 old articles that I have previously updated several months ago, but are they good enough. Due to the nature of the site sometimes I will report on certain news \ features that a web host releases, and sometimes there is very little to write about. These are probably a good example where I feel there is a struggle to write enough or the quality is perhaps lower than something that has more information to report on. I would appreciate some harsh critical view points \ and perhaps suggestions on how to improve my writing style for these kind of generic style posts.
Content Development | | TheWebMastercom0 -
Thoughts on Evan Carmichael for Content Marketing & SEO?
We used to have a lot of success repurposing content through EvanCarmichael.com, but it seems like our articles are being de-indexed there very frequently now. I'd love to hear some others' opinions on Evan Carmichael and how worthwhile it is to keep publishing there. Thanks!
Content Development | | ScottImageWorks0 -
Is there a way to automate finding low quality content on your site?
Hi all, I have a site that was once #1 for many keywords. Fast forward a number of years and I am sinking lower and lower and it really started to sink low from September 2012 and appears to be due to an algo update. How should I go about finding my low quality pages? I am told that I might have pages bringing my entire site down. I deleted heaps of low quality pages but not seeing any improvements (might be a little impatient). Any tips for finding bad content?
Content Development | | BatmanGoonie0 -
How to solve copyright content ?
My website fill of copy paste content , how can i solve this problem?
Content Development | | engmtamous0 -
What Content to Write - Hot Topic or More Niche Related?
Hi, Just for example, say you've got a shoe store but shoes are a non-searched-for topic on the informational side. Say fashion models or teenybopper shoppers are both hot topics. Would you recommend writing an article - one of the site's five >2000 word articles on a hot area of the hot topics? Or would you just stick to shoes topics? If you do write on the hotter topics, how does the shoe store owner write on these - they're out of his area of expertise? Does he need a content writer?
Content Development | | BobGW0 -
Blogger - Multiple partial duplicate content and canonical
In Blogger, have at least three pages produced for each post - main post, archive and tag - each has their own canonical tag - are these considered duplicate content by Google? Not sure the best way to handle this.
Content Development | | holdtheonion0 -
Metrics to measure the quality of content?
When trying to decide what is low quality content, page views & bounce rate are the main indicators I use for pages already on site. But, how do you measure the quality of content that you are trying to produce? Is it entirely subjective?
Content Development | | nicole.healthline0