Is this Duplicate Content?
-
I searched a snippet of one of our Articles (in quotes) and got two results back in Google, one for the article on our site and one for our development/staging site. Does that mean that our development site is getting indexed by Google, even thought we "Disallow:/" in the robots.txt file? Is this a big duplicate content issue?
Thanks
-
I personally have had less of a problem with robots.txt not being followed by Google than I have had developers muck with the robots.txt file. In my particular case, they would update the development site with information from the live site (as we were adding more content to a database on the live site and they were syncing it). They'd copy over the live robots.txt as well, and suddenly the development site was being indexed.
I use Code Monitor for all of my robots.txt files, live or development, so I can be notified when they change. https://www.polepositionweb.com/roi/codemonitor/
I suggest verifying the development site in Google Webmaster Tools. From there, you can request the content to be removed (after you've blocked in robots.txt or used the meta noindex), and you can also use their robots.txt tool to see if there was possibly an error in the file.
-
Yes it does, before no-indexing the page with meta tag, you should put a rel anoical to the right page, as I find the duplicate can hang arounbd in the index for some time. I would let it be crawled again so that the canonical tag can be found before you remove it
-
Yes, you are actually showing 2 shadow copies of your website to Google.
Use a nofollow, noindex tag in the meta. Also, it would be a good idea to have a rel=canonical in place for your testwebsite.(just in case).
-
Yes, that is duplicate content and is an issue. Robots.txt is unreliable at best and there are very stringent guidelines to using it properly.
Instead, use an actual meta tag of noindex! That will solve the issue.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Duplicate product description on the same page
Hello, I've got a really simple a basic question : is it an issue to put an excerpt of a text at the begining of a page, with a "jump to full text" link at the end of the excerpt ? So the first part of the text will be duplicated. A solution would be to hide a part of the text and reveal it with a "read more" button, but we don't want to do that, we want users to be able to read the full text just by scrolling + we want to put an excerpt "above-the-fold" content It's a product page. A second question : we've got duplicate content between our brand category pages, and the related product page, because the presentation text of the brand is shown on the product page + on the brand category page (it's presentations of distilleries, so it's a valuable content for the customer, we want to show it directly on the product page). Do you think it's a big issue ? Or maybe we can think : "ok, there is plenty of other text on those pages, so it won't matter" ?
Content Development | | Colage1 -
Duplicated Terms and Conditions?
Does Google penalise duplicated terms and conditions. I know solicitors use the same legal terms and phrases all the time... Thanks
Content Development | | GaryVictory0 -
One Page Website Blog Content Question
Hi guys, I'm new to the art of SEO and am learning every day from all the fantastic content here, I have a question that I can't find an answer to, hope it doesn't stump you like it has me... I have a one page website (www.neilwilliamsvoiceover.com) that I need to put more content on for SEO purposes but needs to be kept as one page. I've set-up a blog via blogger, and have that on the website but it's in iframe, which I've now discovered is ignored by search engines. So, my question is, is there a way to pull my blog feed into the website and have it recognised by search engines as content for the website? Would I use an RSS feed or feed burner or something else completely?! Thanks for your time and help in advance.
Content Development | | BamMK0 -
On page content and PDF - Dup?
Hi We are writing a useful article which we want to put on our site, but we also want to add it as a pdf which people can download - will this be classed as dup copy?
Content Development | | jj34340 -
Crawl error vs RSS feed content
Hi, I have a BuddyPress Multisite with sites dedicated to specific RSS feeds. I did this to pull in the content for my users. But they generate a 1000s of SEMOZ errors/warnings for dup content, dup titles, missing metatags. etc. So does keeping the content help my site SEO less than the errors? The content is not under my control so I have no way to relate it to my keywords. Any opinions? I can recreate these RSS sites, so I will probably delete them to see what happens. Larry
Content Development | | tishimself0 -
Hosted eCommerce with Outstanding Content Management
Can anyone recommend a hosted eCommerce solution that makes blog/article creation very easy and seamlessly integrates the content into the storefront? If the solution also offered great social media tools, also, that would be great.
Content Development | | DenverKelly0 -
Someone wants to syndicate my content but I'm afraid
I GOT THE FOLLOWING EMAIL: Hi, I visited your site and would like to connect regarding nSphere syndicating your content into our search platform. We are interested in syndicating your content into our network. There’s no cost for this, we ask you to place a module on your site that reads your content and syndicates it into our network of millions of users. You only need to place a simple JavaScript code on your site. I have the code, but I would like to speak or email with you first. This module acts as search tool for your users as well. Can you email me or send me your number and a good day/time to connect? Thank you in advance.
Content Development | | UnderRugSwept
Matthew Mantyla | nSphere
Director of Media Integration I'M AFRAID OF DUPLICATE CONTENT. SHOULD I JUST IGNORE THIS; IS IT SPAM; OR COULD THIS BE AN OPPORTUNITY? I'M THINKING I SHOULD BE SUSPICIOUS OF THE FACT THAT HE DIDN'T EVEN PROVIDE A URL.0 -
Duplicate Content on WordPress Blogs?
We are getting ready to add a WordPress blog to our established website. Our plans are to place it in a subfolder on our website to maximize rank. My question is...Do we need to utilize a Meta Robots WordPress plugin by Yoast or similar so that noindex,follow robots meta tags will prevent search engine indexing of search result pages, subpages and category archives? We want to avoid the dreaded Duplicate Content Error and penalty. Any other great SEO WordPress plugins? Thank you for your time. Brian
Content Development | | gw3seo0