Can I robots.txt an entire site to get rid of Duplicate content?
-
I am in the process of implementing Zendesk and will have two separate Zendesk sites with the same content to serve two separate user groups (for the same product-- B2B and B2C). Zendesk does not allow me the option to changed canonicals (nor meta tags). If I robots.txt one of the Zendesk sites, will that cover me for duplicate content with Google? Is that a good option? Is there a better option.
I will also have to change some of the canonicals on my site (mysite.com) to use the zendesk canonicals (zendesk.mysite.com) to avoid duplicate content. Will I lose ranking by changing the established page canonicals on my site go to the new subdomain (only option offered through Zendesk)?
Thank you.
-
Just disallow in Robots. No need to do anything else.
-Andy
-
What if the site is not live yet?
-
Hi,
I do mean use robots.txt to block crawlers.
What you need to do is first noindex the site in question and then after a period of time, you can disallow it via the robots.txt.
The reason you do it this way is because right now you will have pages from this site indexed in Google - these need to be removed first. You can either do this with the noindex META and wait for Google to spider the site and action all of the noindex requests, or to speed things up, noindex the page and then remove it with Webmaster Tools.
If you don't do this, you are then just blocking the site from Google ever seeing it, so you will probably find that pages remain in the index - which you don't want as this is duplicate content.
-Andy
-
Thank you. I do mean use robots.txt to block crawlers.
-
Hi there!
Just for clarification, I'm really not sure what you mean by "robots.txt-ing" the site. Do you mean, should you use robots.txt to block crawlers from accessing the entire site? That would be fine, if you're not concerned about that site never ranking, ever.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Can I reuse meta description tag on another site?
Hi....lets say i sell widgets and on my site i have redWidgets, blueWidgets and greenWidgets. Each of these widget page's has a meta description tag that seems to help click through. I now have a different site that would like to do a right up about each widget and will dedicate a independent page to each widget. Can i safely provide them with the original meta descriptions (from my original site) for each widget i have on my site to then use on these "right up" pages (as they seem to work well and are informative of the subject matter). I am not sure if this is a seo issue or not. I would think not but would appreciate any input. thank you
On-Page Optimization | | nomad-2023230 -
Site Duplicated despte redirect
Buon pormeriggio from I can smell Whaler Chips Through the window Wetherby,
On-Page Optimization | | Nightwing
When you Google Thakray Medical Museum 2 sites appear in the SERPS, yikes! Now the .org site is no longer hosted & point to the .co.uk site when clicked on but in a nutshell I wantto get rid of the .org site
as illustrated here: http://s216.photobucket.com/user/zymurgy_bucket/media/two-versions-same-website-yikes_zps182e6e12.jpg.html Actions taken so far:
1: Wembaster tools re index request for the .co.uk site
2: Redirect configured to point .org site to the .co.uk What else is left apart from updating the xml site but ultimating i do not want to see the the .org site but it doesnt exist (well id did a few month back but is no longer hosted so i am told) Any insights welcome,
GRazie tanto,
David0 -
Photo Gallery with Duplicate Content and Titles
I have a photo Gallery that is coming up as a lot of Duplicate Titles and Page Content and fixing each photo just isn't possible right now. Should I just block the search engines from indexing them to resolve the errors?
On-Page Optimization | | NeilBelliveau0 -
Is this duplicate content okay?
We have a client who wants to rank locally, nationally and internationally for their products. I wrote a line that goes, "We can ship our products to you whether you’re here in Illinois, nationwide, or international." I added that line after a paragraph or two of unique product description on each of their 30-odd product pages. Will this damage their ranking? I tried researching this but only found full page duplicate content topics. Any advice would be great.
On-Page Optimization | | optimalwebinc0 -
Duplicate content problem
I am having an issue with duplicate content that I can't seem to figure out. I got rid of the www.mydomain.com by modifying the htaccess file but I can't figure out how to fix theproblem of mydomain.com/ and mydomain.com
On-Page Optimization | | ayetti0 -
Checking Duplicate Content
Hi there, We are migrating to a new website, which we are writing lots of new content for the new website. The new website is hosted on a development site which is password protected and so on so that it cannot be indexed. What i would like to know is, how do i check for duplicate content issues out there on the world wide web with the dev site being password protected? Hope this makes sense. Kind Regards,
On-Page Optimization | | Paul780 -
Can embedding videos from other sites help SEO?
I work for an eCommerce site and many of the vendors we get our products from have videos our their site that explain how the products work and the different features of them, ect. We really have done nothing with video marketing, and would like to create some of our own unique videos, but this will be impossible for the vast majority of the products on our site. We know from a content standpoint embedding videos from our vendors to our product pages will be beneficial, but at the same time it shows our competitors and other site visitors where we are getting our products from which we would rather not do. How will this effect us from an SEO standpoint though? Will it help us to have new content, even though it is embedded from another site? Or will it hurt us because it is not unique? Or will it have no effect at all? Thanks for you help!
On-Page Optimization | | ClaytonKendall0 -
Duplicate Content
What I can do to avoid the duplicate content on the index and in the categorys, I cant block my categorys, cause are pages with big autorithy, so what i can do ?
On-Page Optimization | | nafera20