Is this Duplicate Content?
-
I searched a snippet of one of our Articles (in quotes) and got two results back in Google, one for the article on our site and one for our development/staging site. Does that mean that our development site is getting indexed by Google, even thought we "Disallow:/" in the robots.txt file? Is this a big duplicate content issue?
Thanks
-
I personally have had less of a problem with robots.txt not being followed by Google than I have had developers muck with the robots.txt file. In my particular case, they would update the development site with information from the live site (as we were adding more content to a database on the live site and they were syncing it). They'd copy over the live robots.txt as well, and suddenly the development site was being indexed.
I use Code Monitor for all of my robots.txt files, live or development, so I can be notified when they change. https://www.polepositionweb.com/roi/codemonitor/
I suggest verifying the development site in Google Webmaster Tools. From there, you can request the content to be removed (after you've blocked in robots.txt or used the meta noindex), and you can also use their robots.txt tool to see if there was possibly an error in the file.
-
Yes it does, before no-indexing the page with meta tag, you should put a rel anoical to the right page, as I find the duplicate can hang arounbd in the index for some time. I would let it be crawled again so that the canonical tag can be found before you remove it
-
Yes, you are actually showing 2 shadow copies of your website to Google.
Use a nofollow, noindex tag in the meta. Also, it would be a good idea to have a rel=canonical in place for your testwebsite.(just in case).
-
Yes, that is duplicate content and is an issue. Robots.txt is unreliable at best and there are very stringent guidelines to using it properly.
Instead, use an actual meta tag of noindex! That will solve the issue.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why should I avoid publishing off-topic content on my website?
As a fun project, my team wanted to build a mini-food blog based off the lunches we make here at our office -- but we're a software company and, topically, our product has nothing to do with food. Therefore, I suggested that we not publish this content on our website + create a Medium publication instead (this would also help us avoid the headache of creating an entirely new section of our website / potential 404 issues from non-technical editors / etc.) However, I struggled to articulate _why _it's a best practice to only publish relevant content on your website. Is it to help search engines understand what your website is about as an entity? Spam signals?
Content Development | | AsanaOps0 -
Reposting My Original Content
I wanted to know how I should repost my original content from my blog to other sites, obviously quality ones and legacy style content. I don't want get hit with any duplicate content or spam type penalties, but want to further spread some of my content from my blog to get it noticed and drive traffic and leads. Do I submit the posts as is, then put a link to the original source and if so, how many sites can i safely repost the same piece of content too? I am not trying to do anything sneaky, just want it to spread my best content to a good amount high authority sites. I already do social bookmarking and Facebook, twitter etc.
Content Development | | photoseo10 -
How to solve copyright content ?
My website fill of copy paste content , how can i solve this problem?
Content Development | | engmtamous0 -
Finding Good Content Writers
I have a small but growing SEO company. I don't have in house content writers...where is a good place to find good content writers? Please help! Thanks.
Content Development | | ClickIt0 -
Duplicate Page Content & Rel-Canonicals
The SEO Moz duplicate page content tool lists the following URL's as having duplicate content: http://www.savvyboater.com/1988-newer-8-tooth-15-hp-honda-outboard-props.aspx http://www.savvyboater.com/1988-newer-8-tooth-15-hp-honda-outboard-props.aspx?sort=PriceAsc&pi=2 The second URL is the price sorter/second page of the category and contains the following rel-canonical: | http://www.savvyboater.com/1988-newer-8-tooth-15-hp-honda-outboard-props.aspx"> Are we using the rel-canonical correctly in this case? If so, why does it continue to show up as duplicate content in our SEO Moz report? There are over 1,000 URLS listed in the report with the exact same issue. |
Content Development | | ironpac0 -
Is it advisable to have unique pages for different cities/states though there wouldnt be any actual differentiation in the actual content.
Is it advisable to have unique pages for different cities/states though there wouldnt be any actual differentiation in the content. For example should we have separate pages for "hammers in california" & "hammers in new york". The product is same and content more or less the same. The search volume for individual queries is low but collectively makes a large number. The unique title tag automatically will generate traffic. So does it make sense to make 50 such pages. Else is there any way to uniquely target 50 such queries/month/city
Content Development | | DYo0 -
Content Estimates
Are there any SEO Content writers that can create several pages for me on a post- graduate level for a target audience of physicians? Writing should be informative regarding our company and also optimized for keywords. I only have one quote so far from a company outside of SEOMoz and I was hoping to get someone on this site that had some experience in this area. Thank you, Utah Tiger
Content Development | | Boodreaux0 -
Using testimonials to build quality content
I don't see a lot of talk on here about using testimonials to build good content. From what I have seen, the search engines love client testimonials. I haven't found a good service provider for this yet. I'm thinking something combined with Twitter or FB that also allows you to show the content on your site would be great. Can anyone recommend a service like this? I'm checking out a service called, "Quick Vouch" but I haven't tested it yet.
Content Development | | BradBorst0