Is this Duplicate Content?
-
I searched a snippet of one of our Articles (in quotes) and got two results back in Google, one for the article on our site and one for our development/staging site. Does that mean that our development site is getting indexed by Google, even thought we "Disallow:/" in the robots.txt file? Is this a big duplicate content issue?
Thanks
-
I personally have had less of a problem with robots.txt not being followed by Google than I have had developers muck with the robots.txt file. In my particular case, they would update the development site with information from the live site (as we were adding more content to a database on the live site and they were syncing it). They'd copy over the live robots.txt as well, and suddenly the development site was being indexed.
I use Code Monitor for all of my robots.txt files, live or development, so I can be notified when they change. https://www.polepositionweb.com/roi/codemonitor/
I suggest verifying the development site in Google Webmaster Tools. From there, you can request the content to be removed (after you've blocked in robots.txt or used the meta noindex), and you can also use their robots.txt tool to see if there was possibly an error in the file.
-
Yes it does, before no-indexing the page with meta tag, you should put a rel anoical to the right page, as I find the duplicate can hang arounbd in the index for some time. I would let it be crawled again so that the canonical tag can be found before you remove it
-
Yes, you are actually showing 2 shadow copies of your website to Google.
Use a nofollow, noindex tag in the meta. Also, it would be a good idea to have a rel=canonical in place for your testwebsite.(just in case).
-
Yes, that is duplicate content and is an issue. Robots.txt is unreliable at best and there are very stringent guidelines to using it properly.
Instead, use an actual meta tag of noindex! That will solve the issue.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How much do I have to differentiate syndicated content, exactly?
We have about 15-20 articles we'll repurpose on a partner domain (think: media outlet). To avoid duplicate content suspicion, how much exactly do we need to differentiate the content on the second domain? Yea, this is assuming we can't obtain a canonical for whatever reason. I've found some good advice here, but am looking for some quantification. Like: "A sentence/paragraph of introduction at the top of the piece, plus a link back to the original at the end of said introduction ought to do it." Any help is appreciated. Thanks! Tim
Content Development | | Jen_Floyd0 -
Community Discussion - Pitches from content marketers versus publicists: any difference?
Howdy, Moz community! Hope you're all having a fine Friday so far! Tuesday on the blog we featured Samuel Scott's superpowered "Advanced Guide to Online Publicity Campaigns." One interesting tidbit stood out to me as I was reading; the author states: On online marketing websites and blogs, I see pitching often being discussed by "content marketers" as a way to gain shares of and links to one thing or another. They should stop. I receive e-mailed pitches from PR executives and "content marketers" all the time — and I can tell within three seconds which one I'm getting. How? Here is the difference between the two. "Content marketers" pitch me: 1.) To share or link to some random article, and they do so often when
Content Development | | FeliciaCrawford
2.) I have no connection to or interest in the topic at all Publicists pitch me: 1.) To write about an idea because
2.) They already know that I have a connection to or interest in that topic I ignore or delete the pitches from "content marketers." Following the pitches from publishers, I may choose to include their source, study, or idea in some future piece in the publications to which I contribute. Most "link earning" methods are poor imitations of traditional publicity practices. Pitch in a way that will genuinely interest the people who you are contacting. Do not pitch thinly-veiled attempts to get links and shares for you or your clients. I definitely get these emails fairly regularly, but I've never given thought to just what it is that makes me respond positively to some and decline others. So here's my discussion question for the week: What's the distinction for you? Have you noticed that, in your own pitches, you've had a better reception to a certain strategy? Does the "publicist" angle work better in your experience, or have you had plenty of luck with the "content marketer"-type pitch? What do you actually find yourself responding to, in these situations?9 -
Duplicate content penalty
Hi there, I'd like to ensure I avoid a duplicate content penalty and could do with some advice. There is a popular blogger in my industry. I have agreed to add his blog to my website. He currently posts his blog on one of the popular free blogger platforms, and will continue to do this. The issue is that I will be posting duplicate content onto my site and I want to ensure that I do not trigger a google penalty. Is there a simple way form me to inform Google of the original source of the content. My intitial thoughts are: 1. Add a noindex to the Robots.txt file 2. Add a link at the beginning of the article pointing to the original source 3. Adding a rel=canonical tag in the header of each blog entry pointing to the original blog post which resides on a completely different domain. Thanks DBC
Content Development | | DBC011 -
Am I spreading my content & site thin?
I have a video section on my site. Basically I am filtering quality videos for my readers to check out. The videos are pretty much all embedded youtube/vimeo vids. There are a few categories, which are pretty niche-y in relation to my readers. In general they probably aren't seen as too relevant to the overall content on my site... Is it a mistake to keep these videos up? Could they be messing up my rankings since they aren't necessarily in line with the rest of the content on my site?
Content Development | | PedroAndJobu0 -
Duplicate Content Discovery
I was hit with Penguin on April 24th like a ton of bricks. Luckily my cash cow keyword was kept safe and still is today with even an increase in traffic over the year. With some other main keywords I used to rank far I fell off the board on that day. Since then I have been slowly trying to clean things up as much as I know Today I was sitting down with my coffee and Penguin mindset and I decided to use copyscape again to review duplicate content issues and something I noticed which I either didn't before or didn't think was an issue was my footer. In my footer I used a blurb from some other site in my niche a long time ago. Which I discovered they used from one of the main sites in my niche. Anyways I noticed that my footer is what kept coming up as being duplicate content and was always at an overage of 28% according to copyscape. My question is should I be worried about the footer? Is 28% a lot?
Content Development | | cbielich0 -
How much content is needed
I have two clients whose websites have landing pages that feature a number of product links. In order to meet SEO/Google best practices, do I need to have additional content on these specific pages or will the links suffice? (Getpaper is an ecommerce; inpak is not) Any thoughts would be appreciated. http://www.getpaper.com/find-paper/inkjet-plotter-paper/color-bond-21-lb http://www.inpaksystems.com/bag-closing/bag-sewing
Content Development | | TopFloor0 -
Duplicate content - 6 websites, 1 IP. Is the #1 site knocked down too?
Yes I know, running multiple websites on 1 IP isn't smart. 6 Websites with duplicate content on 1 IP is even worse. It's a technical issue we can't solve quickly. Thing is, our #1 website, which has the highest DA and PR, was the first website with all this content. All other websites we're running were launched a few months, and some a few years, later. All content was copied from the #1 website. I'd say the other websites would get knocked down by Google, because they duplicated the content. Google should see that our #1 website was the first that uploaded this content. Therefore our #1 website should rank normally. Questions is: What does Google think of duplicate content when all websites are on 1 IP? Is, or will our #1 website get punished as well?
Content Development | | Webprint0 -
How quickly should one add content?
I'm building a content site (the model is AdSense revenue) around a certain niche, and I'm currently paying for about 6 articles to be contributed per week. I have the capacity to be paying for a lot more articles, however, so I'm wondering what, if any, factors exist to recommend building the site up slowly as opposed to throwing on e.g. 100 articles over the next week? Those I can think of are: 1. Going slowly leaves room for better keyword optimization etc. 2. Google seems to favor aged domains/content, so 100 good articles now certainly isn't as advantageous as 100 articles 2 years from now. All that being said, I still feel like the benefit in terms of traffic of adding more content now - since I can - might outweigh these considerations. Does anyone have any thoughts?
Content Development | | ZakGottlieb710
