Can I robots.txt an entire site to get rid of Duplicate content?
-
I am in the process of implementing Zendesk and will have two separate Zendesk sites with the same content to serve two separate user groups (for the same product-- B2B and B2C). Zendesk does not allow me the option to changed canonicals (nor meta tags). If I robots.txt one of the Zendesk sites, will that cover me for duplicate content with Google? Is that a good option? Is there a better option.
I will also have to change some of the canonicals on my site (mysite.com) to use the zendesk canonicals (zendesk.mysite.com) to avoid duplicate content. Will I lose ranking by changing the established page canonicals on my site go to the new subdomain (only option offered through Zendesk)?
Thank you.
-
Just disallow in Robots. No need to do anything else.
-Andy
-
What if the site is not live yet?
-
Hi,
I do mean use robots.txt to block crawlers.
What you need to do is first noindex the site in question and then after a period of time, you can disallow it via the robots.txt.
The reason you do it this way is because right now you will have pages from this site indexed in Google - these need to be removed first. You can either do this with the noindex META and wait for Google to spider the site and action all of the noindex requests, or to speed things up, noindex the page and then remove it with Webmaster Tools.
If you don't do this, you are then just blocking the site from Google ever seeing it, so you will probably find that pages remain in the index - which you don't want as this is duplicate content.
-Andy
-
Thank you. I do mean use robots.txt to block crawlers.
-
Hi there!
Just for clarification, I'm really not sure what you mean by "robots.txt-ing" the site. Do you mean, should you use robots.txt to block crawlers from accessing the entire site? That would be fine, if you're not concerned about that site never ranking, ever.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How can i increase DA of my site?
hello everyone, I have my 4-month-old blog, PA of the website is 17 yet DA is as yet 5. I don't have the foggiest idea how to expand the DA of the site. It would be ideal if you propose to me how to expand the DA of Site get best jigsaw Thanks
On-Page Optimization | | Salman425521 -
Duplicate content - "Same" profile-information
Hi, I own a casting website with lots of profiles. Some of these profiles only typed in their firstname, email and age, when they registered on the site, and they haven't added more information ever since. From Crawl Diagnostics, I can see that there is "lots" of these profiles, which looks exactly the same (only showing age and firstname), allthought they are not the same. I could add which day the profile were created on the site, to maybe avoid these "duplications". The email will always be hidden. Or, how big an issue is this? Crawl Diagnostics tells me, that there is around 200 of these, and they are "marked" as High Priority. Any ideas on what to do? /Kasper
On-Page Optimization | | KasperGJ0 -
Duplicate content on partner site
I have a trade partner who will be using some of our content on their site. What's the best way to prevent any duplicate content issues? Their plan is to attribute the content to us using rel=author tagging. Would this be sufficient or should I request that they do something else too? Thanks
On-Page Optimization | | ShearingsGroup0 -
Duplicate content with a trailing slash /
Hi, I 've pages like this: A) www.example.com/file/ B) www.example.com/file Just two questions: Does Google see this as duplicate content? Best to 301 redirect B to A? Many thanks Richard PS I read previous threads re the subject, it sounded like there was a bug in SEOMoz but I was not absolutely clear. Apologies if this is going over old ground.
On-Page Optimization | | Richard5550 -
Duplicate products across numerous catalogues on an eCommerce site
We run a site that contains many catalogues aimed at different markets. Products can appear in one or more catalogues. The product copy etc will be on the whole similar maybe even identical across those catalogues. So I'm assuming these will be flagged as duplicates? If so is there a way to fix this other than completely re-writing the copy for each catalogue?
On-Page Optimization | | TTS_Group0 -
How to SEO a website that is being help back by duplicate content?
We have over 20 websites that sell property. Each website is targeted to a different country. People advertise to sell their property. The websites are not getting to page 1 for the terms we want probably because of duplication issues. If we compare one website with another country website on www.duplicatecontent.net we find it is nearly 70% between one and the other. So we trying to understand why this is. If someone wanted to sell a property in Spain we would create an advert for them but rather than putting this on the back-end of the Spain website it goes on a separate website that does on all countries. We have tried to put nofollow tags so that the country specific website gets acknowledgement of being the original website but the rankings for key-terms will not rise and the duplication % remains nearly 70%. Can anyone suggest the best way forward?
On-Page Optimization | | Feily0 -
Duplicate Page Content Question
This article was published on fastcompany.com on March 19th. http://www.fastcompany.com/magazine/164/designing-facebook It did not receive much traffic, so it was re-posted on Co.Design today (March 27th) where it has received significantly more traffic. http://www.fastcodesign.com/1669366/facebook-agrees-the-secret-to-its-future-success-is-design My question is if google will dock us for reprinting/reusing content on another site (even if it is a sister site within the same company). If they do frown on that, is there a proper way to attribute the content to the source material/site (fastcompany.com)?
On-Page Optimization | | DanAsadorian0 -
How Do You Build Good Quality Content on an E-Commerce Site with over 1500 products?
I've been told for a while now that quality is king, and as far as the top level down goes, I have improved the quality of these pages incredibly, working from the landing page down to the categories. Now the problem comes when I know that if I describe each product dynamically with great content, this give the best result, yet doing this on over 1500 products is just a mammoth task for anyone, isn't it? I want a way to have good quality content on each of those pages without writing a million words.... is this possible?
On-Page Optimization | | frank-2443750