How to safely exclude search result pages from Google's index?
-
Hello everyone,
I'm wondering what's the best way to prevent/block search result pages from being indexed by Google. The way search works on my site is that search form generates URLs like:
/index.php?blah-blah-search-results-blahI wanted to block everything of that sort, but how do I do it without blocking /index.php ?
Thanks in advance and have a great day everyone!
-
Hi Louise,
If you can ID the parameters, you can also look at blocking these in Webmaster Tools. This page explains more. As with any blocking of URLs, of course, proceed with caution.
-
I agree that can be effective. The reason I suggested the robots.txt is because Louise mentioned "blocking and preventing" as an objective. Robots.txt are particularly useful in the example where results from a search bar or something of that nature is involved. A NOINDEX, FOLLOW will not prevent bots from getting tired and dizzy, whereas the robots.txt can "block and prevent" bots from crawling certain parameters.
With all of that said, I think it is important to understand whether you need the bots to crawl and not index (in which case Spencer's answer is correct), or if you need to prevent bots from crawling the parameters altogether.
Hope that is more clear
-
I'm not sure that robots.txt is effective when url parameters are involved.
I would just add a meta robots tag to the head section of the search results template:
-
If you are able to identify a url parameter, you may excluded them using robots.txt. Here is a great resource on Robots.txt - http://moz.com/learn/seo/robotstxt
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Which page speed tools do you trust?
We are at a loss. We have been following Google Pagespeed Insights which gives us horrible marks. Nothing we do seems to really make a difference. GTMetrix and Pingdom give us much better scores. We don't know what to do...
Reporting & Analytics | | HashtagHustler0 -
Google is not indexing all URLs
My website have company and events profile from 200 countries. So it does have lots of URL. Earlier in August 2014, Google used to crawl 90% of URLs we submit. Thing goes wrong when we shifted from http to https. We lost traffic. But we are gaining it slowly. Main concern is that, It still does not indexed all submitted URLs. It have crawled merely 8% of all URLs submitted. site address is businessvibes.com Any help would be appreciated.
Reporting & Analytics | | irteam0 -
Webmaster Tools, why does it show 486 pages submitted to web, and only 40 indexed?
I am confused on what a client account shows in WMTs, client account is http://multiview.com. They have a graph showing 486 pages submitted to web, but only 40 are indexed. Also, they recently re-launched, i.e in April 2014, and the new site has about 40 pages indexed.... so I am guessing that the 486 number relates to all the pages that are showing errors in retrieving...i.e. 28 soft 404 errors, 10 access denied errors, 808 not found errors. Does this make sense to explain why there is such a gap between 486 and 40?
Reporting & Analytics | | DianeDP0 -
Tracking in Google Analytics
My site has just recently (or maybe not so recently...) had a great deal of https URL's indexed (I was really only able to find this out thanks to the recent update to the GWT Index Status). It appears that Googlebot picked up an ssl somewhere (I already know where) on my site and then proceeded to crawl and index pages with https rather than http. Since I understand the issue, it should be an easy fix. My question is, does Google Analytics support (track) both http AND https for one site, or would I need to set up two different tracking codes for http and https? I figured that I might as well grab some data from the https pages that are indexed before I try and remove them. I've done a little research on using Groupings/Groups but I figured I would reach out to the MOZ community to see if anyone else has worked with a similar issue. Thanks!
Reporting & Analytics | | GalcoIndustrial0 -
Blog posts not appearing in Search
http://www.themorrisagency.co.uk/blog works well to index my niche terms. Looked at webmaster tools today and since today it has put my indexed pages from 900+ to 11 ??? I have recently added a new site format via a web designer and was all working fine. Now none of my posts are appearing.I have resubmitted site map for 993 pages Is there anything I can do to stop/ prevent this from happening again? and what do you think was the cause? Thanks everyone! Daniel
Reporting & Analytics | | Agentmorris0 -
Setting up Goals in Analytics when no common 'order confirmation' type page
Hi I'm working on a project/client and have asked them for the common url that shows whenever user finishes the purchase process so i cant set it up as a Goal in GA so we can track conversions etc Since site is dynamically generated it does not have such a page. Dev are suggesting setting up a new goal, generate a URL and script and they will append to a similar call. Does this make sense/sound like it would work ? And is it quite easy to do that in Analytics ? Normally i just add a URL as a goal and jobs done, don't know about scripts etc ! Cheers Dan set up a new Goal and generate a URL and script and send that to us and we'll append this to a similar call we already did for this at your request originally
Reporting & Analytics | | Dan-Lawrence0 -
It appears there's a problem with our connection to your Google Analytics account. Please go to your Settings page to update your connection.
I keep getting this error though I have confirmed I have the correct information. Any recommendations?
Reporting & Analytics | | x3oadmin0 -
Does anyone know what's happened to google analytics -> traffic sources -> SEO -> queries many of my accounts are showing a drop to zero in the laste few days
Howdy mozzers It's in the question title really. Zero impressions showing for the last few days on. Multiple accounts Any thought out there
Reporting & Analytics | | Big_Partnership0