What reasons exist to use noindex / robots.txt?
-
Hi everyone. I realise this may appear to be a bit of an obtuse question, but that's only because it is an obtuse question. What I'm after is a cataloguing of opinion - what reasons have SEOs had to implement noindex or add pages to their robots.txt on the sites they manage?
-
Many reasons. You don't want the admin pages of your site indexed, for example. You may not want all of the search queries that people perform on your site search to be indexed. You don't want or need your cart checkout being indexed for an ecommerce site. You don't want a print version and a web version of the same document indexed, so you exclude the print version from being indexed. Your site is in development, and you don't want it being indexed before it is ready.
For robots.txt in particular, some search engines now respect wildcards and you can exclude some session IDs via robots.txt. OSCommerce is real bad about creating session IDs and getting those indexed, then you have tons of different URLs indexed for the same page.
http://www.cogentos.com/bloggers-guide-to-using-robotstxt-and-robots-meta-tags-to-optimise-indexing/ is a post that explains some of the reasons to use robots and no-index on a Wordpress site.
-
There are a couple that come to my mind when i used them working for an agency. I remember one client had some temporary pages that didn't want to get indexed, explaining certain problem with a product at that time. We wanted the page to be live, but didn't want the problems that the product was having to show up in the search engines since it was just temporary.
Also, pages that are targeting same keywords that you dont want to erase or redirect and instead want to keep them live but at the same time you dont want to compete with the other main page. You just block it to the search engines.
Hope this helps
-
I really should have worded my question better.
I'll try again.**What reasons do people have for not wanting their pages show on search results? **
I've got a few reasons of my own, but I'm interested in seeing if there's any I hadn't thought of.
-
For pages you don't want them to show up on search results. =P
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Duplicate content? other issues? using vendor info when selling their prodcuts?
When building content for vendors that we sell their products? best practices? ok, to copy and paste "about us" info? or will that be flagged as duplicate content.
On-Page Optimization | | bakergraphix_yahoo.com0 -
What is the best way to block http://www.site.com/members/...
How do i block http://www.site.com/members/....name/activity/3202 and many more like this from getting spider showing up as duplicate in moz Regards Tai
On-Page Optimization | | Taiger0 -
Word count and ranking/traffic
Hello All, Recently, a client of ours enlisted another firm to redesign their website, www.thepodhotel.com. The new site went live on February 20th. Since that date, we have seen a 30% drop in Google Organic traffic, year-over-year. Even a cursory glance at the site will tell you that there was quite a bit left on the table, from an SEO point of view. However, my question pertains to word count. The word count on the new site is quite a bit lower than the old site. How much value do you feel this has on rankings? Do you feel that, among the many other items that need to be addressed, we should be focusing on creating more copy? I appreciate all of your input. Frank
On-Page Optimization | | FrankSweeney0 -
Using rel="author" on product descriptions
Unique and effective product descriptions can be one of the most expensive investments that go into building a retail website. I would consider them part of the "intellectual property" of a company. These descriptions are also one of the most often pirated types of content. Some product descriptions have been grabbed and used by hundreds of pirates. When people grab your product descriptions it creates a duplicate content problem, it also starts cutting off your long-tail traffic. The discussion topic here is.... Have you heard of anyone using rel="author" on product pages with substantive content? Do you think there is any value in doing that? Is this an abuse of rel="author" Thank you!
On-Page Optimization | | EGOL1 -
How quickly should I change/edit sites to avoid potential over optimization penalty?
Regard today's White Board, I'm wondering how quickly one should start changing title tags, page content, footer links, etc. on sites? I operate a few different sites and some of them definitely have things that need to be changed. Would it be weird or trigger something in Google if all the sites were changed in 24 - 48 hours? Thanks for your help!
On-Page Optimization | | jdrrocks0 -
Negatives to using custom sub domain?
So - being photographers, we have our main website, but also, we use a hosted service for all our client galleries (www.zenfolio.com) So, in effect, we have two websites: Our main informational website Our client gallery/proofing website The client gallery has back links to our main website - so, when people are viewing their gallery, they can easily get back to our main site. We also have thrown a few of our preferred keywords in there for SEO purposes. The gallery has thousands of pages which link back to the main site. So.. the client gallery URL can either be: http://ourbusinessname.zenfolio.com OR we can have it so it uses our own domain, such as: http://gallery.ourbusinessname.com The question is, which domain name will benefit the back links more? Our custom subdomain (which links to our main domain) or, using the Zenfolio domain (which is external to our site). Or, is there no real difference either way? Or.. do I make no sense?
On-Page Optimization | | blitzna100 -
How many keywords / phrases can you SEO for?
Might sound like a bit of a daft question to you pro's but here it goes... I am been doing in house SEO for a company for a short while and since I have been managing it things have moved quickly in the right direction but I find myself limiting our SEO efforts to three key terms. Is this usual? The reason I am doing this is because I find that on page SEO is only really possible with those amount of terms (with regards to title, description, key word density, seo copy, on page links etc). I guess that what I trying to stay is: If I push on page opitimastion to more key terms will it be detrimantal to our current efforts? and Is SEO possible for additional key phrases which are not on page optimised?
On-Page Optimization | | RikkiD220
