I have more pages in my site map being blocked by the robot file than I have being allowed to be crawled. Is Google going to hate me for this?
-
Using some rules to block all pages which start with "copy-of" on my website because people have a bad habit of duplicating new product listings to create our refurbished, surplus etc. listings for those products. To avoid Google seeing these as duplicate pages I've blocked them in the robot file, but of course they are still automatically generated in our sitemap. How bad is this?
-
When you say "people," are you saying your own web team duplicates content to make their job easier? Or am I missing something?...
If that's the case, you really should create unique URL's with unique page titles, product info, etc. That's the correct way to avoid getting hit for duplicate content - don't create it. It seems like what you're doing now is more of a band-aid solution to the problem.
I'd consider that even though creating unique content in situations like this can seem daunting and/or be more expensive, there's probably huge long-term gains to made if you do it right.
-
It is not bad, just not best practices because Google will still index the URL's if they are mentioned on other pages. Just to quote them:
"While Google won't crawl or index the content of pages blocked by robots.txt, we may still index the URLs if we find them on other pages on the web. As a result, the URL of the page and, potentially, other publicly available information..."
What I would do instead is either use rel="canonical" or 301 redirects. I hope that helps.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unique Pages with Thin Content vs. One Page with Lots of Content
Is there anyone who can give me a definitive answer on which of the following situations is preferable from an SEO standpoint for the services section of a website? 1. Many unique and targeted service pages with the primary keyword in the URL, Title tag and H1 - but with the tradeoff of having thin content on the page (i.e. 100 words of content or less). 2. One large service page listing all services in the content. Primary keyword for URL, title tag and H1 would be something like "(company name) services" and each service would be in the H2 title. In this case, there is lots of content on the page. Yes, the ideal situation would be to beef up content for each unique pages, but we have found that this isn't always an option based on the amount of time a client has dedicated to a project.
On-Page Optimization | | RCDesign741 -
Does no-follow for pages affect site ranking?
Hey, I have a question. On my site, it's divided into the main site and the blog is in a subfolder of same domain. Within the main site (same domain), there are MANY checkout pages and other internal pages we use though all with "NO FOLLOW" on each. Despite it having "NO FOLLOW", will it affect our blog rankings in any way or domain ranking?"
On-Page Optimization | | Mirian0 -
Site wide content like "why choose us" just above the footer on every single page
Hi Guys, I know that is not good having any kind of duplicate content on your site, but SEO is above all "competition", so I have to see what my competitor are doing to find the best way to outrank them. So this is my question: is it good or not having site wide content like "why choose us" just above the footer on every single page? At the moment, I can see many - too many - of my client competitors having the "Why choose us" as site wide content above the footer. The funny thing they don't use a couple of sentences, they have placed many words and 10/20 internal links, in other words, they have enough stuff to put down a stand alone page. What do you think: this is just a bad SEO practice or it may work, as I can see so many sites ranking well with this kind of piece of junk on each page. I am not going to recommend this to my client, but as am trying to detail every decision I make showing what the competitors are currently doing, my concern is that my client finds it and therefore will ask to have the same shiny piece of garbage above the footer. Thanks, Pierpaolo
On-Page Optimization | | madcow780 -
Why does Google pick a low priority page on my site?
Hi Guys. One of my pages ranks quite well for "mid year diaries 14-15" on Google. The problem is it's a really specific product page (A4, Hardback, day-to-a-page diary I think). It would be much better for the user to land on our mid-year diaries category, not really deep into the site. Why is Google prioritizing this product page over our general 'mid year diaries' category? Especially when the category would relate to the search more accurately? I work for TOAD diaries and I think our page rank is 10 for this search. Eagerly awaiting some insight 🙂 Thanks in advance everyone! Isaac.
On-Page Optimization | | isaac6630 -
When You Add a Robots.txt file to a website to block certain URLs, do they disappear from Google's index?
I have seen several websites recently that have have far too many webpages indexed by Google, because for each blog post they publish, Google might index the following: www.mywebsite.com/blog/title-of-post www.mywebsite.com/blog/tag/tag1 www.mywebsite.com/blog/tag/tag2 www.mywebsite.com/blog/category/categoryA etc My question is: if you add a robots.txt file that tells Google NOT to index pages in the "tag" and "category" folder, does that mean that the previously indexed pages will eventually disappear from Google's index? Or does it just mean that newly created pages won't get added to the index? Or does it mean nothing at all? thanks for any insight!
On-Page Optimization | | williammarlow0 -
Google picking up old pages
I recently redesigned a site that had all the keywords it was ranking for going to the home page. Now I have specific pages for each of these keywords but I'm seeing the home page (not the page that, if I do an on page optimization by hand in MOZ gives me an A rating) showing up in the auto reports (assuming pages Google sees for these keywords related to the url) as F's. They're all pointing to the home page. I've redirected the old index.html home page to the new but I suspect the reason is actually these pages (were) ranking for these terms (though none too well - all but one were not in the top 50 and one was 45) because these rankings are all dropping as well. I'm at a loss, with the site replaced, as to how to correct this and tell Google these keyword phrases all have their own pages now. I've dug through this forum and the only applicable answer I can see would be to add these phases to the home page (where they all rank for now) with anchored links to their new (A rated by Moz for these terms when I hand enter them) singular pages? Or is it just a waiting game?
On-Page Optimization | | adworksofboca0 -
Opinions please on Duplicate page titles & too many on-page links warnings.-
Hello folks, I'm a total SEO newbe but totally enjoying
On-Page Optimization | | CSC
using SEOmoz to learn more. We have ecommerce sites and the 1st crawl flags – as appears typical too many on-page links. We display up to 20 products (each with three links!)
and I’m trying to push to have fewer but meeting resistance from colleagues.
We have links duplicated all over the site believing it eases navigation. My question is just how critical is the number of products displayed
and the resulting volume of links to SEO results? Also we currently have collections of products displayed
across several pages which of course have the same page title and this is flagged
as a duplication error. I wonder if product auto-scrolling help as this means only a certain number of products are displayed at one time on one page thus reducing links and the need for duplicate page titles? My superiors are resisting change (perhaps nervous of spoiling
what already works) and I need to know where to direct my persuasive powers! Many thanks in anticipation, Spence0 -
Authority of a page
What factors contribute towards the authority of a page ? No. of links to a page ?
On-Page Optimization | | seoug_20050