Is this Duplicate Content?
-
I searched a snippet of one of our Articles (in quotes) and got two results back in Google, one for the article on our site and one for our development/staging site. Does that mean that our development site is getting indexed by Google, even thought we "Disallow:/" in the robots.txt file? Is this a big duplicate content issue?
Thanks
-
I personally have had less of a problem with robots.txt not being followed by Google than I have had developers muck with the robots.txt file. In my particular case, they would update the development site with information from the live site (as we were adding more content to a database on the live site and they were syncing it). They'd copy over the live robots.txt as well, and suddenly the development site was being indexed.
I use Code Monitor for all of my robots.txt files, live or development, so I can be notified when they change. https://www.polepositionweb.com/roi/codemonitor/
I suggest verifying the development site in Google Webmaster Tools. From there, you can request the content to be removed (after you've blocked in robots.txt or used the meta noindex), and you can also use their robots.txt tool to see if there was possibly an error in the file.
-
Yes it does, before no-indexing the page with meta tag, you should put a rel anoical to the right page, as I find the duplicate can hang arounbd in the index for some time. I would let it be crawled again so that the canonical tag can be found before you remove it
-
Yes, you are actually showing 2 shadow copies of your website to Google.
Use a nofollow, noindex tag in the meta. Also, it would be a good idea to have a rel=canonical in place for your testwebsite.(just in case).
-
Yes, that is duplicate content and is an issue. Robots.txt is unreliable at best and there are very stringent guidelines to using it properly.
Instead, use an actual meta tag of noindex! That will solve the issue.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Simple Blog Content Question
Which is better? To write my own blog post or, (with permission) use other high DA content on my blog. I'll probably do both, but I'm very curious as to what the search engines prefer or which is better for seo. Thanks in advance!
Content Development | | MissThumann0 -
Same content but translated. Penalization?
Hi There, I’ve got a question. There are two website that are under the same proprietor but must stay not related (different brand, different IP, different country, different language). The question is: Does google penalize one of the site if I entirely translate the content from site 1 to site2? Thank you very much for you input 😉
Content Development | | Midleton0 -
Multiple listings for the same product - how to avoid duplication?
Hi everyone, We are working on an e-commerce site that has three different physical stores that each hold different products in stock. A listing is created on the website for each product in each location but if the same product is in all three stores the images and description are duplicate content We can't have one product page and tag locations to it as this would then require keeping two different stock systems updated, but is there a way to avoid the three pages being duplicates of each other? Many thanks for the advice!
Content Development | | A_Q0 -
What are the effects of posting on a blog site using copied content from your site if you source the site on the blog site?
I have noticed that some of my pages from my site are posted to different blog sites and directory submissions they are word for word what is on my site but at the bottom of the page it does list my site as the source and there are links in the text back to my site. Is this going to negatively effect my rankings or is it ok since my site is sourced in the article?
Content Development | | steve2150 -
Same content on site blog as a separate blog. Will unpublishing on one blog evade duplicate content issues?
I just discovered my client was posting the same content as the site I'm working on for him on a separate blog. I don't want to run into duplicate content issues. Both are Wordpress sites. Will it suffice to simply unpublish duplicate entries on the other blog and leave the posts as drafts?
Content Development | | locallyrank0 -
Questions about reorganizing a website's content structure
I'm working with a publisher who is considering reorganizing the content on their site, which up to now has been presented more or less as a portal site for a variety of segments in a particular business industry. One scenario that is on the table is to remove some content sections altogether and republish them under their own unique domains. It's important to note that this reorganization would be part of a new brand strategy. So my first question is whether this is one of those avoid-at-all-costs scenarios? My second question is if we decide to procede, what kind of time could it take for these new domains to generate the same level of search traffic they are currently pulling in on the portal site? Thanks!
Content Development | | accessintel0 -
Duplicate content on forums?
I am creating a forum. I am concerned that when I create the forum, users will copy content from other places to post onto my forum. How negative/bad is this in terms of google eyes? I am concerned when people copy press releases and re-post it to the forum. Should I make a rule that all content must be typed and not copied? Or is a little copying okay?
Content Development | | sseibel0 -
Define: Good Content
I am curious to hear what you guys consider to be the characteristics of good content and in which order if you have a preference. Here are a few I can think of: Informative (you can learn something new) Substantial (enough of it and thorough) Complete (doesn't give half-baked information or ideas) Unique (not regurgitated original content) Helpful (practical actionable information) Visual (content complemented by media) Referenced (claims made are substantiated through citations) Entertaining (or otherwise emotional, e.g. surprising, sad, shocking, controversial) Formatted (easy to read and follow) Timely (right content at the right time, applies for news) Professional (writing style, grammar, spelling and sentence structure) Can you add to this list?
Content Development | | Dan-Petrovic1