I have more pages in my site map being blocked by the robot file than I have being allowed to be crawled. Is Google going to hate me for this?
-
Using some rules to block all pages which start with "copy-of" on my website because people have a bad habit of duplicating new product listings to create our refurbished, surplus etc. listings for those products. To avoid Google seeing these as duplicate pages I've blocked them in the robot file, but of course they are still automatically generated in our sitemap. How bad is this?
-
When you say "people," are you saying your own web team duplicates content to make their job easier? Or am I missing something?...
If that's the case, you really should create unique URL's with unique page titles, product info, etc. That's the correct way to avoid getting hit for duplicate content - don't create it. It seems like what you're doing now is more of a band-aid solution to the problem.
I'd consider that even though creating unique content in situations like this can seem daunting and/or be more expensive, there's probably huge long-term gains to made if you do it right.
-
It is not bad, just not best practices because Google will still index the URL's if they are mentioned on other pages. Just to quote them:
"While Google won't crawl or index the content of pages blocked by robots.txt, we may still index the URLs if we find them on other pages on the web. As a result, the URL of the page and, potentially, other publicly available information..."
What I would do instead is either use rel="canonical" or 301 redirects. I hope that helps.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google ranking content for phrases that don't exist on-page
I am experiencing an issue with negative keywords, but the “negative” keyword in question isn’t truly negative and is required within the content – the problem is that Google is ranking pages for inaccurate phrases that don’t exist on the page. To explain, this product page (as one of many examples) - https://www.scamblermusic.com/albums/royalty-free-rock-music/ - is optimised for “Royalty free rock music” and it gets a Moz grade of 100. “Royalty free” is the most accurate description of the music (I optimised for “royalty free” instead of “royalty-free” (including a hyphen) because of improved search volume), and there is just one reference to the term “copyrighted” towards the foot of the page – this term is relevant because I need to make the point that the music is licensed, not sold, and the licensee pays for the right to use the music but does not own it (as it remains copyrighted). It turns out however that I appear to need to treat “copyrighted” almost as a negative term because Google isn’t accurately ranking the content. Despite excellent optimisation for “Royalty free rock music” and only one single reference of “copyrighted” within the copy, I am seeing this page (and other album genres) wrongly rank for the following search terms: “free rock music”
On-Page Optimization | | JCN-SBWD
“Copyright free rock music"
“Uncopyrighted rock music”
“Non copyrighted rock music” I understand that pages might rank for “free rock music” because it is part of the “Royalty free rock music” optimisation, what I can’t get my head around is why the page (and similar product pages) are ranking for “Copyright free”, “Uncopyrighted music” and “Non copyrighted music”. “Uncopyrighted” and “Non copyrighted” don’t exist anywhere within the copy or source code – why would Google consider it helpful to rank a page for a search term that doesn’t exist as a complete phrase within the content? By the same logic the page should also wrongly rank for “Skylark rock music” or “Pretzel rock music” as the words “Skylark” and “Pretzel” also feature just once within the content and therefore should generate completely inaccurate results too. To me this demonstrates just how poor Google is when it comes to understanding relevant content and optimization - it's taking part of an optimized term and combining it with just one other single-use word and then inappropriately ranking the page for that completely made up phrase. It’s one thing to misinterpret one reference of the term “copyrighted” and something else entirely to rank a page for completely made up terms such as “Uncopyrighted” and “Non copyrighted”. It almost makes me think that I’ve got a better chance of accurately ranking content if I buy a goat, shove a cigar up its backside, and sacrifice it in the name of the great god Google! Any advice (about wrongly attributed negative keywords, not goat sacrifice ) would be most welcome.0 -
Is it bad to include google Maps in footer?
We have 5 locations and we were thinking about including a map for each location in the footer. These would be set-up as no-follow links. They could potentially enhance user experience but it also increases size of footer. Right now there are just basic links to pages (sitemap, terms, etc), contact info, social links, and contact form. If we did the maps it would also include link to the individual location pages. Not sure if we are doing too much in footer or need to just keep it basic. Thanks for the help!
On-Page Optimization | | Restore0 -
Google indexing page differently
Does google index an interal page differently depending on whether you are using a FULL url (including domain) or just a relative link? Also, is it possible that using a full URL (http://mysite.com/page.html) causes the browser to "ping" the server differently than just having the href linked to using relative links (/page.html) Could this cause server or firewall perfomance issues?
On-Page Optimization | | WebRiverGroup0 -
"City page" links in footer of home page: Spammy?
Is listing a bunch of links to city pages in the footer of a home page considered "spammy" to Google? (ie- Chicago Alarms, Illinois Alarms, Naperville Alarms, etc.) What are the negative affects this might have on ranking, if any?
On-Page Optimization | | MChi0 -
How google handle with Title when is invisible on the page?
I would like to display H1 Title tag invisible on the homepage. I set up Title colour same colour as background colour. How google handle with this cloaking? What should I to set up in the style.css?
On-Page Optimization | | joeko0 -
Do I need a robots meta tag on the homepage of my site?
Is it recommended to include on the homepage of your site site? I would like Google to index and follow my site. I am using WordPress and noticed my homepage is not including this meta tag, therefore wondering if I should include it?
On-Page Optimization | | asc760 -
Why has my site gone from 3rd to 21st in Google...HELP
Hi there, Web Address: www.websitedesign-miltonkeynes.com Okay I stared my own web development business about 8 - 9 months ago and decided that I was going to target a small number of local keywords that were relevant to my business and geographical area. So I decided to target the keywords below and and started to get good traffic, I never have bought links and have not copied any content although I know i should try and add more content to my website. mk web design, software Milton Keynes, web design Milton Keynes, web design mk, web designer Milton Keynes, web designers Milton Keynes & website design milton Keynes However recently my website has dropped massively in the SERP's and wanted to know what has caused this (i know Penguin) and what I can do to improve. I have listed below my rankings to show you the drop: 24th April 2012 15th May 2012 mk web design 3 11 software milton keynes 1 8 web design milton Keynes 12 50 web design mk 5 53 web designer milton keynes 10 0 web designers milton keynes 6 37 website design milton keynes 3 21 I decided to change my homepage Page Title a week ago to make sound less spammy but this has made no difference and wanted some help on what has happened so i do not do this again and what I can do to improve. Thanks in advance. Darren Bowden
On-Page Optimization | | Tarqs0 -
Should our social network put all of our member profiles in the site map?
We're working on some changes to our site and want to add a site map file to give some hints to search engines on our member profiles. Our site has millions of member profiles. I want to ensure that Google indexes them. Is there a benefit to submitting millions of profiles across multiple site maps? Is it better to pick a sample of those and submit them? Most member profiles have links to other member profiles. Is there any benefit to taking a single site map of 50k links and then changing it at some frequency so that all of the member profiles are eventually indexed? Thanks.
On-Page Optimization | | JoeCotellese810