Is this Duplicate Content?
-
I searched a snippet of one of our Articles (in quotes) and got two results back in Google, one for the article on our site and one for our development/staging site. Does that mean that our development site is getting indexed by Google, even thought we "Disallow:/" in the robots.txt file? Is this a big duplicate content issue?
Thanks
-
I personally have had less of a problem with robots.txt not being followed by Google than I have had developers muck with the robots.txt file. In my particular case, they would update the development site with information from the live site (as we were adding more content to a database on the live site and they were syncing it). They'd copy over the live robots.txt as well, and suddenly the development site was being indexed.
I use Code Monitor for all of my robots.txt files, live or development, so I can be notified when they change. https://www.polepositionweb.com/roi/codemonitor/
I suggest verifying the development site in Google Webmaster Tools. From there, you can request the content to be removed (after you've blocked in robots.txt or used the meta noindex), and you can also use their robots.txt tool to see if there was possibly an error in the file.
-
Yes it does, before no-indexing the page with meta tag, you should put a rel anoical to the right page, as I find the duplicate can hang arounbd in the index for some time. I would let it be crawled again so that the canonical tag can be found before you remove it
-
Yes, you are actually showing 2 shadow copies of your website to Google.
Use a nofollow, noindex tag in the meta. Also, it would be a good idea to have a rel=canonical in place for your testwebsite.(just in case).
-
Yes, that is duplicate content and is an issue. Robots.txt is unreliable at best and there are very stringent guidelines to using it properly.
Instead, use an actual meta tag of noindex! That will solve the issue.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Hi friends, I have a question. I want to know how Google detects the author of a content.
Hi friends, I have a question. I want to know how Google detects the author of a content.
Content Development | | Alopii0 -
Internal blog with history and some SEO value versus new external blogs with specialized content?
We operate a blog inside a folder on our site and considering the launch of 4 highly focused blogs with specialized content which are now categories on the internal blog. Wondering if there is more value in using the external new blogs or just keep growing the internal blog content. Does fact that the internal blog is buried amongst millions of pages have any impact if we want the content indexed and value given to the links from the blog content to our main site pages.
Content Development | | CondoRich0 -
When Somebody Copies your content What should you do?
I wrote an article in 2011 A Brief History of Benjamin Moore Paint on my website ( I am a painting contractor). It is a content creators win; sited as an authoritative article on wikipedia. Benjamin Moore Paint company has copied my content on their own corporate website; a deep page. Is this hurting, helping, or neutral? bX1kFQg,639HqZE bX1kFQg,639HqZE#1
Content Development | | johnshearer0 -
Site Content Review Please!
I m looking for someone who can review my site and let me about quality of content on my site. Can anyone suggest / know who I can talk to about this ? Nick
Content Development | | orion680 -
Duplicate Content
I am wondering what is the best way to show google that there is duplicate content on the page. for example on our product pages they are unique content except we give the same guarantee and promise on every product providing some duplicate content. What is the best way to fix this issue?
Content Development | | DoRM0 -
Fresh content ideas for a static site?
I have an ecommerce site. My home page is set-up just as I want it. I'm not looking to redo it or change my site to a blog. Just looking for some new, different, SEO friendly ideas or concepts to keep it "fresh".
Content Development | | VictorVC0 -
How often should content be updated
With all of Google's recent algo updates (or ranking updates, whatever they're calling it now), we've obviously been looking into changing our content strategy and shifting it from quantity to quality. How often would you say is ideal for website content updates? i.e. should we be updating once a month? Once every couple of months? This isn't a blog - just a regular services-oriented site. My take on it is that it should be as often as organically possible - and that means something different for everyone. At the same time, we want Google coming back frequently to crawl the site. Thanks!
Content Development | | eyecarepro0 -
If you have the best sales on the weekends, when is the best day to post the content?
The Client has the best sales on the weekends and his writing a weekly blog post to help the overall site rank better and hopefully drive more sales. When is the best time to post this content? On Monday so that it hopefully gets indexed by the weekend? or On Friday so that we can leverage twitter and Facebook to increase sales.
Content Development | | anchorwave0