Is this Duplicate Content?
-
I searched a snippet of one of our Articles (in quotes) and got two results back in Google, one for the article on our site and one for our development/staging site. Does that mean that our development site is getting indexed by Google, even thought we "Disallow:/" in the robots.txt file? Is this a big duplicate content issue?
Thanks
-
I personally have had less of a problem with robots.txt not being followed by Google than I have had developers muck with the robots.txt file. In my particular case, they would update the development site with information from the live site (as we were adding more content to a database on the live site and they were syncing it). They'd copy over the live robots.txt as well, and suddenly the development site was being indexed.
I use Code Monitor for all of my robots.txt files, live or development, so I can be notified when they change. https://www.polepositionweb.com/roi/codemonitor/
I suggest verifying the development site in Google Webmaster Tools. From there, you can request the content to be removed (after you've blocked in robots.txt or used the meta noindex), and you can also use their robots.txt tool to see if there was possibly an error in the file.
-
Yes it does, before no-indexing the page with meta tag, you should put a rel anoical to the right page, as I find the duplicate can hang arounbd in the index for some time. I would let it be crawled again so that the canonical tag can be found before you remove it
-
Yes, you are actually showing 2 shadow copies of your website to Google.
Use a nofollow, noindex tag in the meta. Also, it would be a good idea to have a rel=canonical in place for your testwebsite.(just in case).
-
Yes, that is duplicate content and is an issue. Robots.txt is unreliable at best and there are very stringent guidelines to using it properly.
Instead, use an actual meta tag of noindex! That will solve the issue.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Duplicate content
Dear community, We have 15 product specific landing pages. They all share a block called "Why invest in VanEck ETFs?", see e.g., https://www.vaneck.com/de/en/mining-etf https://www.vaneck.com/de/en/space-etf/ https://www.vaneck.com/de/en/esports-etf/
Content Development | | marketing-europe
Can this lead to SEO penalization because of duplicate content?0 -
Does google penalize you if you post content in french and english on a website
I'm trying to encourage content editors to only post content in either English or French. For example we have a French press release but the team are wanting it on our site in French and English. I thought this would fall under duplicate content rules. Does google penalize you if you post content in French and English on a website?
Content Development | | EstherBrice0 -
Content Creation For Blog & Ranking Locally
Hey Everyone, I'm trying to rank for specific long tail key words such as: lower back pain treatment exercises lower back surgery options herniated disc exercises and etc. My question is: If I create a blog or article on these key words and integrate content and video within it and on youtube, will these blog posts come up locally when someone searches it within my area?
Content Development | | backinmotion1231 -
Can I use expired domain content on My Blog
Hello Expert, I wanted to know, can I use expired domain content on my blog channel. I have done many searches on google but couldn't find a satisfactory answer. Please help me to find out this.
Content Development | | vijay77960 -
How important is exact keyword match in content for SEO in 2018?
Is it necessary to have the exact keyword which you are trying to rank for within the first sentence of the content on your web page? For example if we are selling men's shoes, is it ok to have have the meta title and h1 on page say men's shoes and just talk about shoes within the first sentence of the content on the page. Is it obvious to search engines that this page is selling men's shoes even if you don't have the word men's shoes in that first sentence or is it necessary to have the exact keywords which you are trying to rank for written there?
Content Development | | whiteonlySEO0 -
Wordpress Blog Pages, Duplicate Title Tag
Anyone have any experience in fixing the duplicate Title tag on a Wordpress blog multiple pages Basically the title tag remains the same on the pages /Blog/ /Blog/Page/2/ /Blog/Page/3/ My good friend Yoast Plugin doesn't seem to of resolved this (Unless i have missed something?) I don't really see this to be effecting anything and wouldn't of through it would either, but it would be nice to not see the notification within Moz site crawls and campaigns etc, its more of a cosmetic problem Any solutions ? Thanks James
Content Development | | Antony_Towle0 -
Content Syndication Service
Am curious to get the forum's opinion on content syndication services like SYNND . Has anyone tried them or any other content syndication networks. Does the forum have any recommendations for good quality syndication networks that can be used to distribute content to quality sources.
Content Development | | SEO5Team0 -
Duplicate content and Facebook
If i have content on my site and the same content duplicated on my facebook pages, will google treat this as duplicate content? At the moment when i copy and paste a line of text from the content on my site Facebook is returned first.
Content Development | | Turkey0