I have more pages in my site map being blocked by the robot file than I have being allowed to be crawled. Is Google going to hate me for this?
-
Using some rules to block all pages which start with "copy-of" on my website because people have a bad habit of duplicating new product listings to create our refurbished, surplus etc. listings for those products. To avoid Google seeing these as duplicate pages I've blocked them in the robot file, but of course they are still automatically generated in our sitemap. How bad is this?
-
When you say "people," are you saying your own web team duplicates content to make their job easier? Or am I missing something?...
If that's the case, you really should create unique URL's with unique page titles, product info, etc. That's the correct way to avoid getting hit for duplicate content - don't create it. It seems like what you're doing now is more of a band-aid solution to the problem.
I'd consider that even though creating unique content in situations like this can seem daunting and/or be more expensive, there's probably huge long-term gains to made if you do it right.
-
It is not bad, just not best practices because Google will still index the URL's if they are mentioned on other pages. Just to quote them:
"While Google won't crawl or index the content of pages blocked by robots.txt, we may still index the URLs if we find them on other pages on the web. As a result, the URL of the page and, potentially, other publicly available information..."
What I would do instead is either use rel="canonical" or 301 redirects. I hope that helps.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Word Count - Content site vs ecommerce site
Hi there, what are your thoughts on word count for a content site vs. an ecommerce site. A lot of content sites have no problem pushing out 500+ words per page, which for me is a decent amount to help you get traction. However on ecommerce sites, a lot of the time the product description only needs to be sub-100 words and the total word count on the page comes in at under 300 words, a lot of that could be considered duplicate. So what are your views? Do ecommerce sites still need to have a high word count on the product description page to rank better?
On-Page Optimization | | Bee1590 -
How should i optimize this page
Hi, i am having major problems in optimizing this page as it is a magazine site. On normal sites i have no problem in optimizing the page to get the correct keywords to come up in the search engines but since the upgrade and also because it is a magazine site, i am having problems on how i should do this. my site is www.in2town.co.uk and i am trying to optimize the page for the following keywords lifestyle magazine online magazine lifestyle news Life and Style articles healthy lifestyle i am trying to make sure that google knows what the magazine is about, as i know have dropped down the rankings since the upgrade and for lifestyle magazine we were number one in google for such a long time but now we are on page 9 and this is our home page. we are seeing sites that have hardly any content ranking above us for this keyword i have a small intro which i have just put in the past few days at the top and we have a welcome in the middle which is here. Welcome to In2town Lifestyle Magazine Our Lifestyle Magazine is a fresh, innovative and vibrant online magazine offering you the best in health,fitness and life & style features, as well as modern lifestyle, beauty, fashion, personal finance and entertainment. Over the years In2town Lifestyle Magazine has established a reputation for quality articles and informed lifestyle and health features thanks to our experienced team of editorial professionals. By reading our online Lifestyle Magazine, you will be able to enjoy the interesting mix of entertainment features, health and lifestyle news as well as finding out what is happening in the celebrity world. We are always happy to hear from our readers, if you have lifestyle news or a story that you feel our readers would be interested in then please do contact us. xxxxxxxx but i would like to get rid of that section as i am going to put the latest articles there. any advice on how to sort this mess out would be great
On-Page Optimization | | ClaireH-1848860 -
Will Google handle "this not that" pages differently?
If you create pages about "try keyword1 not keyword2" will there be any barriers to getting the pages ranked for keyword2? Example: You have furnished rental units in a small town, and you offer nightly/weekly rentals. You want to rank for "town hotel" since you offer the same service as a hotel. Since you're not really a hotel, you create a page called "Better than a hotel: Town nightly rental units". Anyone know if Google has an algorithm to detect this (they would have to detect the meaning of the words you were using and know that you were promoting something other than a hotel) and determine you're not really relevant to "town hotel" and not rank you well? I think they probably do not, as I've seen things like Google Adsense Alternatives articles ranking well for the term Google Adsense, or Boycott Godaddy sites ranking well for the term godaddy. But I would like to hear any evidence or facts others know of.
On-Page Optimization | | AdamThompson0 -
How much SEO value does a fashion site get from bolting text onto the bottom of home page? Does the value compensate for cluttering up a page focused on an iconic image?
Getting ready to launch a completely redesigned site for a fashion designer. Since it is a fashion site, visitors do not need text to describe what the site is about., We are weighing three options: 1) clean design with no text (just images and navigational links), 2) bolting on a couple of sentences of text at the bottom of the page to signal keyword terms to the search engines, 3) following the lead of the top ranking site in the category and adding lots of text to the bottom of the page. Do the SEO benefits justify cluttering up the design by bolting text onto the bottom of the home page, and if so, how many characters of text seem to be the minimum to be effective?
On-Page Optimization | | RandyP0 -
Can Page Authority of a site be higher than its Domain Authority?
I own a website called Takeyourtips.com. While doing a search no Google, I found that the page authority of the home page (31) is higher than the domain authority (23). I was wondering if it's really possible because my understanding was page authority of a page is determined by its domain authority. Therefore, it the domain authority of a website is 23, none of its page could have a higher page authority. Plus, upon consulting an SEO expert, I was told that neither Domain Authority or Page Authority of a page carries any importance as far as higher ranking of a website is concerned. Is this true? Thanks in advance for the answers. Cheers, Sushant large
On-Page Optimization | | suskanchan0 -
Optimally, how many times should the key word or phrase you are targeting for a particular page be mentioned or appear on that page?
Our marketing team is debating how many times the key phrase on each of our web store's product pages should include the word/phrase we are trying to be competitive with. Can you advise?
On-Page Optimization | | Glynlyon0 -
Does Google respect User-agent rules in robots.txt?
We want to use an inline linking tool (LinkSmart) to cross link between a few key content types on our online news site. LinkSmart uses a bot to establish the linking. The issue: There are millions of pages on our site that we don't want LinkSmart to spider and process for cross linking. LinkSmart suggested setting a noindex tag on the pages we don't want them to process, and that we target the rule to their specific user agent. I have concerns. We don't want to inadvertently block search engine access to those millions of pages. I've seen googlebot ignore nofollow rules set at the page level. Does it ever arbitrarily obey rules that it's been directed to ignore? Can you quantify the level of risk in setting user-agent-specific nofollow tags on pages we want search engines to crawl, but that we want LinkSmart to ignore?
On-Page Optimization | | lzhao0 -
Can I have a strong brand category page and a strong product page?
It seems Google base and other Comparison Shopping Engines like to see the brand in the product name. But, on my category page for that brand, website optimizer tells me including the brand name with each product is cannabilizes links. For example; I have a page for jewelerABC with 20 pieces of jewelry listed as well as original content about jewelerABC. I do not currently name these products as xyz by jewelerABC. This page comes up nicely in the serps. But in Google base The top listings for jewelry by jewelerABC seem to have every product named xyz by jewelerABC or JewelerABC xyzs. What is the best way to optimize.for both? Stephen
On-Page Optimization | | stephenfishman0