Is this Duplicate Content?
-
I searched a snippet of one of our Articles (in quotes) and got two results back in Google, one for the article on our site and one for our development/staging site. Does that mean that our development site is getting indexed by Google, even thought we "Disallow:/" in the robots.txt file? Is this a big duplicate content issue?
Thanks
-
I personally have had less of a problem with robots.txt not being followed by Google than I have had developers muck with the robots.txt file. In my particular case, they would update the development site with information from the live site (as we were adding more content to a database on the live site and they were syncing it). They'd copy over the live robots.txt as well, and suddenly the development site was being indexed.
I use Code Monitor for all of my robots.txt files, live or development, so I can be notified when they change. https://www.polepositionweb.com/roi/codemonitor/
I suggest verifying the development site in Google Webmaster Tools. From there, you can request the content to be removed (after you've blocked in robots.txt or used the meta noindex), and you can also use their robots.txt tool to see if there was possibly an error in the file.
-
Yes it does, before no-indexing the page with meta tag, you should put a rel anoical to the right page, as I find the duplicate can hang arounbd in the index for some time. I would let it be crawled again so that the canonical tag can be found before you remove it
-
Yes, you are actually showing 2 shadow copies of your website to Google.
Use a nofollow, noindex tag in the meta. Also, it would be a good idea to have a rel=canonical in place for your testwebsite.(just in case).
-
Yes, that is duplicate content and is an issue. Robots.txt is unreliable at best and there are very stringent guidelines to using it properly.
Instead, use an actual meta tag of noindex! That will solve the issue.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Content: Best Blogs Article
Hello, For an Ecommerce site, I think a good way to get known is to write a "Best X Blogs" article, where X is a topic in your industry, and then letting the people you link to know about the article. I got the idea from a Mozinar. My question is, how close does the X from above have to be in your niche? For example, if your product is running shoes can you write a "best athlete blogs" article? I'm worried about them reading the article, then leaving. In some smaller niches the topics closest to the product don't have much in the way of blogs out there. So how close to your niche does the Best X Blogs topic have to be?
Content Development | | BobGW0 -
Same content on site blog as a separate blog. Will unpublishing on one blog evade duplicate content issues?
I just discovered my client was posting the same content as the site I'm working on for him on a separate blog. I don't want to run into duplicate content issues. Both are Wordpress sites. Will it suffice to simply unpublish duplicate entries on the other blog and leave the posts as drafts?
Content Development | | locallyrank0 -
Does a blog appearing in diff. categories cause duplicate content?
Having a particular blog appearing in different categories or under different tags on a site- does it cause duplicate content? If yes, why ?
Content Development | | Personnel_Concept0 -
Content Marketing Strategy for a Solar Panel Shop
Hi everybody Everybody's talking about content marketing strategies these days and the videos from MozCon has definitely been a great inspiration for me. I have a customer who sells solar panels (wiki) here in Denmark and they are more or less hooked on the idea of investing long term. And this is where I wanna ask you guys for ideas to this strategy. I'm open to anything from overall guidelines to specific ideas on content. I hope you will chip in, thanks a lot 🙂
Content Development | | loevgaard0 -
Promoting blog content
I've created a pinterest board, which kind of serves as a blog also as it tells people how they can save money on their heating bills this winter. Does anyone have any suggestions on how to promote it as a backlinking opportunity? Would it be good to also feature the content on our website? Guest blogging? Any suggestions would be welcomed. Thank you.
Content Development | | AAttias0 -
RSS feeds with dup content and titles
Hi, For my Buddypress site I use a tool to create sites with RSS feeds. Each site is for a different feed, but the number of dup tiles and content is running in the thousands. I've been trying to reduce the dups, but have begun to think there is more trouble from such content than benefit. Should I dump the content or ignore the errors flagged by SEOMOZ? Any ideas if thes RSS feed dups are hurting my BuddyPress site? Any suggestions in general about how to eliminate such dupe for a Buddypress Site, eg. the activity log. Larry
Content Development | | tishimself0 -
Blog and Website = 2 different URL's - Is it WORTH to merge content on to one site
Good day Mozzers! A friend of mine recently asked for my help in regards to online marketing. While getting familiar with his online presence, I realized that he has a blog hosted under a completely different URL Main Site = http://pardons.org/ (page rank 4)
Content Development | | vip4service
Blog = http://pardons.wordpress.com/ (page rank 3) What I am battling with is whether or not he should take all of the blog content he has, and merge it on to his main site. It has over 280+ blog posts spanning over a few years, so there is A LOT of content that could benefit his main site. However is it worth it, or should he continue to run everything as 2 different sites? Also, of you suggest moving the content over, what would be the best way to do it in your opinion? He currently has links on his blog TO his main site, so there is a little bit of link juice there, but with a average of 300 views a day, he only get about 10 clicks to his main site from the blog. Thanks a ton for your help!0