Can I robots.txt an entire site to get rid of Duplicate content?
-
I am in the process of implementing Zendesk and will have two separate Zendesk sites with the same content to serve two separate user groups (for the same product-- B2B and B2C). Zendesk does not allow me the option to changed canonicals (nor meta tags). If I robots.txt one of the Zendesk sites, will that cover me for duplicate content with Google? Is that a good option? Is there a better option.
I will also have to change some of the canonicals on my site (mysite.com) to use the zendesk canonicals (zendesk.mysite.com) to avoid duplicate content. Will I lose ranking by changing the established page canonicals on my site go to the new subdomain (only option offered through Zendesk)?
Thank you.
-
Just disallow in Robots. No need to do anything else.
-Andy
-
What if the site is not live yet?
-
Hi,
I do mean use robots.txt to block crawlers.
What you need to do is first noindex the site in question and then after a period of time, you can disallow it via the robots.txt.
The reason you do it this way is because right now you will have pages from this site indexed in Google - these need to be removed first. You can either do this with the noindex META and wait for Google to spider the site and action all of the noindex requests, or to speed things up, noindex the page and then remove it with Webmaster Tools.
If you don't do this, you are then just blocking the site from Google ever seeing it, so you will probably find that pages remain in the index - which you don't want as this is duplicate content.
-Andy
-
Thank you. I do mean use robots.txt to block crawlers.
-
Hi there!
Just for clarification, I'm really not sure what you mean by "robots.txt-ing" the site. Do you mean, should you use robots.txt to block crawlers from accessing the entire site? That would be fine, if you're not concerned about that site never ranking, ever.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Can't get my preferred URL, how much does it matter?
Hi guys. I'm building a new site at the moment - seen a solid SEO opportunity for my work. I'm a producer engineer, specialising in mixing and mastering, so i'm creating a site for online mixing services. After a bit of keyword research I decided that "online mixing" was the best, most relevant and high volume term to go for. Ideally i'd like my home page to be www.onlinemixing.com (or something similar) but alas! It's been taken, as well as all the variations (like switching words, hypens etc) How much does this matter form an SEO point of view? E.g - For the search term "online mixing" would - www.onlinemixing-signalchain.co.uk be much worse than -www.onlinemixing.co.uk? Or am I sweating the small stuff? Any thoughts would be greatly appreciated. Isaac.
On-Page Optimization | | isaac6630 -
Why Moz is showing Duplicate Page Content Issues?
We have a Career Section on our website. For each job post, there is a separate link of "Apply Job". Now Moz's Crawl Diagnostic is showing Duplicate page content for such URLs. Here are two such URLs: http://tiny.cc/em9nyw http://tiny.cc/bq9nyw Can any one please suggest on this? Thanks
On-Page Optimization | | chandman0 -
Issue: Duplicate Page Content
For duplicate page content, how different should pages be? For example, I have seven locations and on each location page, we offer a discount. The discounts are the same currently and open into a pop-up window. So it looks something like this: mysite.com/locationA/dicount mysite.com/locationB/discount mysite.com/locationX/discount The pages are identical. Should I change the verbiage on each page or let it be? I noticed that our organic search rankings have dropped since our site upgrade and this is one item that SEOMOZ has noted. Thanks! DHO
On-Page Optimization | | DougHoltOnline0 -
When Adding content to the site. Should I use the same keyword term on each page or select a secondary keyword to focus on?
I have created a site www.autoinsurancefremontca.com. The index page is SEO for the key term auto insurance fremont ca. I want to add more content on another page of this site. Should I have that page also SEO'd for the same keyword or should I pick another keyword to focus on?
On-Page Optimization | | Greenpeak0 -
Exponentially Increasing Duplicate Content On Blogs
Most of the clients that I pick up are either new to SEO best practices, or have worked with sketchy SEO providers in the past, who did little more than build spammy links. Most of them have deployed little if any on-site SEO best practices, and early on I spend a lot of time fixing canonical and duplicate content issues alla 301 redirects. Using SEOMOZ, however, I see a lot of duplicate content issues with blogs that live on the sites I work on. With every new blog article we publish, more duplicate content builds up. I feel like duplicate content on blogs grows exponentially, because every time you write a blog article, it exists provisionally on the blog homepage, the article link, a category page, maybe a tag page, and an author page. I have a two-part question: Is duplicate content like this a problem for a blog -- and for the website that the blog lives on? Are search engines able to parse out that this isn't really duplicate content? If it is a problem, how would you go about solving it? Thanks in advance!
On-Page Optimization | | RCNOnlineMarketing0 -
Issue: Duplicate Page Content (index.htm)
I get an error of "**Issue:**Duplicate Page Content" for the following pages in the SEOMOZ Crawl Diagnostics. But these pages are the same one! Duhhhh.... Is there a way to hide this false error? http://www.stdtime.com/ http://www.stdtime.com/index.htm BTW, I also get "**Issue:**Duplicate Page Title" for this page. Another false error...
On-Page Optimization | | raywhite0 -
I have one page on my site... but still get duplicate name and content errors.
i have only the index.html page. my domain has a permanent 301 to the root. why am i getting duplicate problems? i only have one page the index .html???
On-Page Optimization | | one4u2see0 -
Meta description tags & title tags duplicate content?
Although meta description tags have no impact on rankings - can there be a negative effect from having many duplicate meta title + description tags with just 1-2 words switched out?
On-Page Optimization | | qlkasdjfw0