What reasons exist to use noindex / robots.txt?
-
Hi everyone. I realise this may appear to be a bit of an obtuse question, but that's only because it is an obtuse question. What I'm after is a cataloguing of opinion - what reasons have SEOs had to implement noindex or add pages to their robots.txt on the sites they manage?
-
Many reasons. You don't want the admin pages of your site indexed, for example. You may not want all of the search queries that people perform on your site search to be indexed. You don't want or need your cart checkout being indexed for an ecommerce site. You don't want a print version and a web version of the same document indexed, so you exclude the print version from being indexed. Your site is in development, and you don't want it being indexed before it is ready.
For robots.txt in particular, some search engines now respect wildcards and you can exclude some session IDs via robots.txt. OSCommerce is real bad about creating session IDs and getting those indexed, then you have tons of different URLs indexed for the same page.
http://www.cogentos.com/bloggers-guide-to-using-robotstxt-and-robots-meta-tags-to-optimise-indexing/ is a post that explains some of the reasons to use robots and no-index on a Wordpress site.
-
There are a couple that come to my mind when i used them working for an agency. I remember one client had some temporary pages that didn't want to get indexed, explaining certain problem with a product at that time. We wanted the page to be live, but didn't want the problems that the product was having to show up in the search engines since it was just temporary.
Also, pages that are targeting same keywords that you dont want to erase or redirect and instead want to keep them live but at the same time you dont want to compete with the other main page. You just block it to the search engines.
Hope this helps
-
I really should have worded my question better.
I'll try again.
**What reasons do people have for not wanting their pages show on search results? **
I've got a few reasons of my own, but I'm interested in seeing if there's any I hadn't thought of.
-
For pages you don't want them to show up on search results. =P
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Using Bold text for keywords
Hello I am updating an old e-commerce website of mine and many keywords are in bold - shall I remove the bold tag or keep them there? This is for SEO.
On-Page Optimization | | xdunningx0 -
Htaccess redirect, from /year/month to /blog
I am trying to make some redirects so we don't lose that SEO juice. I am trying to move our blog structure from:
On-Page Optimization | | opiates
http://www.domain.com/2015/09/title-of-blog to:
http://www.domain.com/blog/title-of-blog I need to do redirects in htaccess from the old structure to the new structure but I can't seem to get it working properly. Here is what I have thus far. <code>RewriteEngine on RewriteBase / RewriteRule ^(.0-9)\/^(.0-9)\$ $1/ [R=301,L]</code> Any suggestions?0 -
Index / Monthly Click Number
Hi,
On-Page Optimization | | HypermediaSystems
This is a general question, so sorry in advance if inappropriate. Once I was told, in large scale EC / Forum Site,
the following number should be around 1,
and if it is below 1, it is a good sign ... Google Indexed Page Number / Monthly ( 30days ) Click Number I was told this is just a general idea, and real world situation varies, then
if you don't have any standard, this could be a start. (not dogmatic rules, just reference) Does this sounds about right? or do you have any other formula? I was tasked to do the site wide SEO, and diagnose the general state of SEO-wellness/fitness..
and right now, the number is 1.5, so I am about to report we can do more to get more SERP presence or something... If you guys point me relevant blog article / Q&A forum, I would really appreciate. Thanks!0 -
How long should I leave an existing web page up after a 301 redirect?
I've been reading through a few of blog posts here on moz and can't seem to find the answer to these two questions: How long should I leave an existing page up after a 301 redirect? The page old page is no longer needed but has pretty high page authority. If I take the old page down—the one that I'm redirecting from—immediately after I set up the 301 redirect, will link juice still be passed to the new page? My second question is, right now, on my index.html page I have both a 301 redirect and a rel canonical tag in the head. They were both put in place to redirect and pass link equity respectively. I did this a couple years back after someone recommended that I do both just to be safe, but from what I've gathered reading the articles here on moz is that your supposed to pick one or the other depending on whether or not it's permanent. Should I remove the rel conanical tag or would it be better to just leave it be?
On-Page Optimization | | ScottMcPherson0 -
On page/site - Images & bold text
As I understand it on a page where you are going for the keywords 'Metal gates', you have an image titled 'Metal gates' (Lets say MG for Metal gates to save on my typing!) with the alt tag being 'MG' Now as I also understand it, if an image is also used as a link the Alt tag is also the anchor text? So, I have a page on Metal gates using the above methods, from this page I have image links to a metal driveway gates page (as well as text links), would I be best using an image titled 'Metal driveway gates' with the alt text as ' Metal driveway gates' to link to my metal driveway gates page (if you follow me)? Also on the image question, what if you have say six images (that are all links) on the metal gates page, how would you title the images and alt tags? Variations on Metal gates? Or using the keyword on the page that the link points to? (I'm not looking to keyword stuff but want to get it right! ) Also is it important to have your on page keywords in bold still? i ask because I've bolded text on my pages but not the main keywords - I've bolded text that I want people to see, but do not want to take away any power from my keywords if you follow me?
On-Page Optimization | | Jon-C0 -
Keyword Cannibalization/stuffing on an ecommerce category page
Hi, Whats the best way to tackle e-commerce category pages? If you have, say, a category showing 30 pairs of socks, and each of the sock products in the lists has a 'view more' link, a link from the product name and a link from the thumbnail. Naturally each of those links should be the product name - sprinkled with a slight variation, a preceding 'View more on [product name]' or superseded with the shop name, so you dont end up with complete duplicate link titles, you get the idea. But you suddenly end up with 90 instances of links with title tags containing 'socks', which ultimately lead to keyword stuffing/cannibalization - especially as you then move to another category with, say, sports socks showing 40 products and therefore 120 link titles also with the word 'socks' Thought on a postcard please? Thanks Tom
On-Page Optimization | | pretige120 -
Moving a site as a folder of existing one
We have the following case: Site A with domain authority 65 and 2,750 linking root domains (139K total links) and Site B with domain authority 68 and 1,336 linking root domains (38K total links) There's a suggestion to move Site A as a folder of Site B so it becomes something like this: domainB.com/domainA using 301 redirects for the existing domain A. We'd like to better understand what such move will result in. I imagine at first Site A will drop in rankings, but after that will it be better for it to be under Site B domain? Also, moving all the pages a step behind in the URL path may slightly affect their rankings. What do you think? Would you move Site A or leave it as a separate domain? I understand there can't be an exact estimation what will happen, but I'll appreciate your thoughts.
On-Page Optimization | | lgrozeva0 -
SEO Change/Version Control
Hi. Can anyone suggest apps that might be applied for SEO change/version control? I've tried using Excel and Word in the past but both fall short when dealing with multiple client sites. Nothing suitable pops up so far. Specifically I want to be able to log any on-page changes that might impact SEO for historical analysis and possible roll-back (manual) eg change date, page title, headings, url, anchor text, alt image txt, body copy etc. The ability to have one view that can be filtered in various ways including by client and by url would be ideal. Maybe this is something SEOmoz is considering for Pro campaign management? Surely there's demand... Thanks Andy
On-Page Optimization | | Bluesnapper0