Is this Duplicate Content?
-
I searched a snippet of one of our Articles (in quotes) and got two results back in Google, one for the article on our site and one for our development/staging site. Does that mean that our development site is getting indexed by Google, even thought we "Disallow:/" in the robots.txt file? Is this a big duplicate content issue?
Thanks
-
I personally have had less of a problem with robots.txt not being followed by Google than I have had developers muck with the robots.txt file. In my particular case, they would update the development site with information from the live site (as we were adding more content to a database on the live site and they were syncing it). They'd copy over the live robots.txt as well, and suddenly the development site was being indexed.
I use Code Monitor for all of my robots.txt files, live or development, so I can be notified when they change. https://www.polepositionweb.com/roi/codemonitor/
I suggest verifying the development site in Google Webmaster Tools. From there, you can request the content to be removed (after you've blocked in robots.txt or used the meta noindex), and you can also use their robots.txt tool to see if there was possibly an error in the file.
-
Yes it does, before no-indexing the page with meta tag, you should put a rel anoical to the right page, as I find the duplicate can hang arounbd in the index for some time. I would let it be crawled again so that the canonical tag can be found before you remove it
-
Yes, you are actually showing 2 shadow copies of your website to Google.
Use a nofollow, noindex tag in the meta. Also, it would be a good idea to have a rel=canonical in place for your testwebsite.(just in case).
-
Yes, that is duplicate content and is an issue. Robots.txt is unreliable at best and there are very stringent guidelines to using it properly.
Instead, use an actual meta tag of noindex! That will solve the issue.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What are the effects of posting on a blog site using copied content from your site if you source the site on the blog site?
I have noticed that some of my pages from my site are posted to different blog sites and directory submissions they are word for word what is on my site but at the bottom of the page it does list my site as the source and there are links in the text back to my site. Is this going to negatively effect my rankings or is it ok since my site is sourced in the article?
Content Development | | steve2150 -
Is there a way to set up a wordpress site so that the content is changed based on a location?
I have a site I am building that has information that shifts based on weather conditions and location. Is there a way to have information sort based on a users geographic location? I am trying to figure out the best way to do this.
Content Development | | Ron_McCabe0 -
Correction Duplicate Page Title Problems for a Blog
EDITED: To just focus on the issue at hand. I am trying to figure out the SEO rules instead of just working on the content. Please bear with me. I am adept technically. I just do not know the rules of the SEO process or even some of the termology. So I’m trying to attack problems one at time. Today’s problem – **Duplicate Page Titles ** We evidently have thousands of Duplicate Page Titles. We are using Joomla 2.5 & Easyblog. Our sitemap is automated from XML Sitemap Easyblog takes the title of the sites and uses it for a name of the summary pages. We post 5 blog items per page and all the names are the same. http://www.OursiteName.com/?start=5 Page Title = Site Name http://www.OursiteName.com/?start=10 Page Title = Site Name A similar thing happens on the sorting by Author or Category etc etc. Basically non-duplicate pages are looking like duplicates. What is the best practice / approach? Using the Robot.txt or XML Sitemap to tell Google not to crawl these pages? Writing a script or edit the Easyblog code to edit the 2000 duplicate Page Titles? Other thoughts?
Content Development | | Romana0 -
What are the best content writer sites?
Hi, I'm doing some work on a new blog and wondered if anyone could recommend some low cost content writers? I have only justed started researching this service, so any advice the SEOmoz community could give would be grately appreciated. Thanks in advance.
Content Development | | RBH0 -
Crawl error vs RSS feed content
Hi, I have a BuddyPress Multisite with sites dedicated to specific RSS feeds. I did this to pull in the content for my users. But they generate a 1000s of SEMOZ errors/warnings for dup content, dup titles, missing metatags. etc. So does keeping the content help my site SEO less than the errors? The content is not under my control so I have no way to relate it to my keywords. Any opinions? I can recreate these RSS sites, so I will probably delete them to see what happens. Larry
Content Development | | tishimself0 -
How quickly should one add content?
I'm building a content site (the model is AdSense revenue) around a certain niche, and I'm currently paying for about 6 articles to be contributed per week. I have the capacity to be paying for a lot more articles, however, so I'm wondering what, if any, factors exist to recommend building the site up slowly as opposed to throwing on e.g. 100 articles over the next week? Those I can think of are: 1. Going slowly leaves room for better keyword optimization etc. 2. Google seems to favor aged domains/content, so 100 good articles now certainly isn't as advantageous as 100 articles 2 years from now. All that being said, I still feel like the benefit in terms of traffic of adding more content now - since I can - might outweigh these considerations. Does anyone have any thoughts?
Content Development | | ZakGottlieb710 -
Tools to Eval Blog Content - Rate your Fav tool
Ok, so I know that is has been covered in depth and at the risk of being sent to “google it!” (Which I have done with no success) I thought that I would ask your opinions on the topic. What are the best content marketing evaluation tools? By this I am specifically referring to tools that evaluate the content of Blogs, etc and not the performance of the blog, etc. I’m eager to hear your thoughts of what works and if you care to share what tools did not. Thanks
Content Development | | Questionmana0