Is this Duplicate Content?
-
I searched a snippet of one of our Articles (in quotes) and got two results back in Google, one for the article on our site and one for our development/staging site. Does that mean that our development site is getting indexed by Google, even thought we "Disallow:/" in the robots.txt file? Is this a big duplicate content issue?
Thanks
-
I personally have had less of a problem with robots.txt not being followed by Google than I have had developers muck with the robots.txt file. In my particular case, they would update the development site with information from the live site (as we were adding more content to a database on the live site and they were syncing it). They'd copy over the live robots.txt as well, and suddenly the development site was being indexed.
I use Code Monitor for all of my robots.txt files, live or development, so I can be notified when they change. https://www.polepositionweb.com/roi/codemonitor/
I suggest verifying the development site in Google Webmaster Tools. From there, you can request the content to be removed (after you've blocked in robots.txt or used the meta noindex), and you can also use their robots.txt tool to see if there was possibly an error in the file.
-
Yes it does, before no-indexing the page with meta tag, you should put a rel anoical to the right page, as I find the duplicate can hang arounbd in the index for some time. I would let it be crawled again so that the canonical tag can be found before you remove it
-
Yes, you are actually showing 2 shadow copies of your website to Google.
Use a nofollow, noindex tag in the meta. Also, it would be a good idea to have a rel=canonical in place for your testwebsite.(just in case).
-
Yes, that is duplicate content and is an issue. Robots.txt is unreliable at best and there are very stringent guidelines to using it properly.
Instead, use an actual meta tag of noindex! That will solve the issue.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Internal blog with history and some SEO value versus new external blogs with specialized content?
We operate a blog inside a folder on our site and considering the launch of 4 highly focused blogs with specialized content which are now categories on the internal blog. Wondering if there is more value in using the external new blogs or just keep growing the internal blog content. Does fact that the internal blog is buried amongst millions of pages have any impact if we want the content indexed and value given to the links from the blog content to our main site pages.
Content Development | | CondoRich0 -
Tool to identify duplicated content on other sites
Hi does anyone know of a tool that could be used to identify if a site is using our content without permission? Thanks
Content Development | | turismodevino10 -
Community Discussion - Are data AND storytelling the missing ingredients for successful content marketing efforts?
Data is an important element of content marketing. Storytelling, too, gets readers' attention and has been shown to be instrumental in prospects and customers forming strong connections to brands. But using data and storytelling helps produce some of the strongest content there is to be shared, says Nichole Elizabeth DeMeré in her latest YouMoz article, Here’s How to Combine Storytelling and Data to Produce Persuasive Content. What are your thoughts? Think data and storytelling work best separately? Read the post are share your thoughts below. RS
Content Development | | ronell-smith2 -
How Many Words on Page for Content When Optimizing a K.W.
I want to hired a writer to create content. When optimizing a keyword on a page, how many words (minimum) should I have on that content. Some writer use ''Word Count'' when fixing a price for text, before asking a writer, I need to specified ''How many word'' to included in the content. Thank you,
Content Development | | BigBlaze2050 -
What are your favourite tools for discoving popular content?
Since looking around for popular content discovery tools I came across a review about a tool called PostRank which seemed ideal until I learned Google had bought it and shut it down already 😞 So far I have been using Google reader and Topsy to discover popular content in my niches but I am guessing there are a whole bunch of other tools out there that may work even better - please do let me know your favourites!
Content Development | | Clicksjim0 -
Is this dulplicate content causing me problems
Hi, i am going to try and explain this as best as possible as i am not used to working with k2 that much. What i have as an example is a page which is here http://www.in2town.co.uk/Cheap-Flights-Ryanair-News This page is news about ryanair now because i am working in joomla and i am working with k2, and i am also making each landing page a front page, i am having problems. For people who do not use joomla i will explain. For the ryanair page which is a front page, i have to make a category and that category for all my content to go into is called ryaniar-news. Now i have the page above which is for ryanair news but the problem i have got is, because i have a category it brings up all the content as a blog format. so for example here is the category section in blog format. http://www.in2town.co.uk/news/ryaniar-news Now i have managed to make sure that the articles are not showing but it brings up the title of the pages and i am worried that this will be seen as duplicate content and will damage the main page of http://www.in2town.co.uk/Cheap-Flights-Ryanair-News I may have set everything up wrong i am not sure but if i have i would be glad if someone can tell me the right way to do it. I do not like the blog format and that is why i use a front page set up so i can have my layout the way i want it. Can anyone please explain if my set up will damage the main ryanair news page or what i need to do to sort it out please.
Content Development | | ClaireH-1848860 -
Can you link build without adding any content to the website?
I am an agency-side SEO who has a few different SEO clients. A couple of them absolutely refuse to add any content to their site - no blog, no articles, no link bait, nothing. They have resisted efforts for any content to be placed on their site - whether it is written by us, them, or a third party. They just don't see the value in it, despite my attempts to persuade them. As a result, these websites are just brochure websites. What are your options for link building in this situation? If content is the foundation of white hat link building, what do you do if the client refuses to add content to their site? All help gratefully received! Thanks
Content Development | | kevinmorley0 -
Duplicating Articles/Blogs
Two years ago, some articles and blogs were written and submitted to certain tech and news sites relevant to our field. Will it be fine to submit those to other sites now. It is our content only and not some one else, but would it be penalty to submit the copy to other article sites whereever those articles/content are relevant today also?
Content Development | | sayeed0