I have more pages in my site map being blocked by the robot file than I have being allowed to be crawled. Is Google going to hate me for this?
-
Using some rules to block all pages which start with "copy-of" on my website because people have a bad habit of duplicating new product listings to create our refurbished, surplus etc. listings for those products. To avoid Google seeing these as duplicate pages I've blocked them in the robot file, but of course they are still automatically generated in our sitemap. How bad is this?
-
When you say "people," are you saying your own web team duplicates content to make their job easier? Or am I missing something?...
If that's the case, you really should create unique URL's with unique page titles, product info, etc. That's the correct way to avoid getting hit for duplicate content - don't create it. It seems like what you're doing now is more of a band-aid solution to the problem.
I'd consider that even though creating unique content in situations like this can seem daunting and/or be more expensive, there's probably huge long-term gains to made if you do it right.
-
It is not bad, just not best practices because Google will still index the URL's if they are mentioned on other pages. Just to quote them:
"While Google won't crawl or index the content of pages blocked by robots.txt, we may still index the URLs if we find them on other pages on the web. As a result, the URL of the page and, potentially, other publicly available information..."
What I would do instead is either use rel="canonical" or 301 redirects. I hope that helps.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Landing page separate from product page
Hello there, I have a wordpress website with a woocommerce plugin. I have 4 landing pages that describe my products and at the end of the pages, I have a CTA to my product page. is it bad for SEO? my website: https://relationadviser.ir
On-Page Optimization | | Aaron.be1 -
Contact pages coming up for keywords above landing pages
I have two examples of contact pages coming up over designated landing pages Keyword: Nickel Alloys for www.neonickel.com Keyword: Artificial Grass for www.artificialgrass4u.co.uk Is there anyway I can stop this happening?
On-Page Optimization | | icansee0 -
Blocking Subdomain from Google Crawl and Index
Hey everybody, how is it going? I have a simple question, that i need answered. I have a main domain, lets call it domain.com. Recently our company will launch a series of promotions for which we will use cname subdomains, i.e try.domain.com, or buy.domain.com. They will serve a commercial objective, nothing more. What is the best way to block such domains from being indexed in Google, also from counting as a subdomain from the domain.com. Robots.txt, No-follow, etc? Hope to hear from you, Best Regards,
On-Page Optimization | | JesusD3 -
How to Fix Google Webmasters Soft 404 Errors for Wordpress Site?
I am getting Soft 404 errors in my Google Webmasters Tools don't know how to fix it the site in on WordPress CMS the site is "http://appdictions.com/" i am getting errors in the http://appdictions.com/members section...suggestion to fix the issue will be appreciated..
On-Page Optimization | | preferati0 -
Google's Page Layout Algorythm
It seems that Google have been or will penalizing websites with too many ads above the fold. Is it me or Google's search result layout is a perfect example of what NOT to do?
On-Page Optimization | | sbrault741 -
What precaution should we take to change the default page of the site
For some reason we wanted to change the default page of my site from example.com to example.com/default.aspx. We will be using 301 redirection to get the back link benefits. Do we need to make any changes in webmaster tool and sitemap too??
On-Page Optimization | | CyrilWilson0 -
Lead With Branded Keywords or Descriptive Keywords in Page Title for (Niche) Site?
Our site is hingeheads.com, and our products and product catalog are unique in two ways. For one our product is not something that people are generally aware of, and secondly our entire product catalog consists of different variations of the same product. **Catalog Overview: **http://hingeheads.com/collections/all Product Example: http://hingeheads.com/products/dolphin I keep wondering if it is better to lead the title with "branded keywords" [1] or with "descriptive keywords" [2]? Dolphin HingeHead | Unique Home Decor & Gift Idea | HingeHeads Dolphin Decor Accessories & Unique Gift Ideas | HingeHeads I am currently going with the second solution, but I am always wondering if that's the right/better solution. I am curious to hear feedback from people who have more experience with this than I do. How would you structure the title for our product pages? Thanks! Kai
On-Page Optimization | | hingeheads0 -
Why Does SEOMOZ Crawl show that i have 5,769 pages with Duplicate Content
Hello... I'm trying to do some analysis on my site (http://goo.gl/JgK1e) and SEOMOZ Crawl Diagnostics is telling me that I have 5,769 pages with duplicate content. Can someone, anyone, please help me understand: how does SEOMOZ determine if i have duplicate content Is it correct ? Are there really that many pages of duplicate content How do i fix this, if true <---- ** Most important ** Thanks in advance for any help!!
On-Page Optimization | | Prime850