How to safely exclude search result pages from Google's index?
-
Hello everyone,
I'm wondering what's the best way to prevent/block search result pages from being indexed by Google. The way search works on my site is that search form generates URLs like:
/index.php?blah-blah-search-results-blahI wanted to block everything of that sort, but how do I do it without blocking /index.php ?
Thanks in advance and have a great day everyone!
-
Hi Louise,
If you can ID the parameters, you can also look at blocking these in Webmaster Tools. This page explains more. As with any blocking of URLs, of course, proceed with caution.
-
I agree that can be effective. The reason I suggested the robots.txt is because Louise mentioned "blocking and preventing" as an objective. Robots.txt are particularly useful in the example where results from a search bar or something of that nature is involved. A NOINDEX, FOLLOW will not prevent bots from getting tired and dizzy, whereas the robots.txt can "block and prevent" bots from crawling certain parameters.
With all of that said, I think it is important to understand whether you need the bots to crawl and not index (in which case Spencer's answer is correct), or if you need to prevent bots from crawling the parameters altogether.
Hope that is more clear
-
I'm not sure that robots.txt is effective when url parameters are involved.
I would just add a meta robots tag to the head section of the search results template:
-
If you are able to identify a url parameter, you may excluded them using robots.txt. Here is a great resource on Robots.txt - http://moz.com/learn/seo/robotstxt
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google Analytics not Working on all pages despite many attempts
Hey gang, So this problem has been going on for months. None of our landing pages track, but all of our blog pages do. I've tried GA plugins, (we are on wordpress) and just now even put the code in the theme header script. Still to no avail. I just checked right now, and the blog pages have the global GA code, but now I noticed not even the home page has the code! Could it be something to do with my theme? I tried to pay a guy on upwork to fix but he couldn't fix it either. Thank you for you help!
Reporting & Analytics | | Meier0 -
Search console Search Analytics devices not showing mobile and tablet data since July 29th, have anyone noticed that too?
If you filter for devices in the search analytics at search console you get that from July 29th all the data is tagged as desktop and mobile and tablet have no data from that date. I see that for all my websites I have search console for, any input on that?
Reporting & Analytics | | amirbt0 -
Google Analytics to sub domains
Hi, I have a site xyz.com and two separate sites on sub domains xyz.com/abc and xyz.com/def. What's the best way to go about with GA such that I can get all the data in the same place. Should I use the GA code for xyz.com on sub domains as well? Or should I create separate profiles?
Reporting & Analytics | | mayanksaxena0 -
Google Analytics Code
We have a quick question about our Google Analytics code: We recently updated to a re-marketing Analytics code, and some of our traffic numbers seem to be off by a bit. I used the Google Tag Assistant Chrome Extension today, and noticed that it's finding our old Analytics code on our pages, but it's coming up as an error due to "no HTTP Response". I am attempting to remove this code from the website, but it is nowhere to be found in the HTML coding. Only the current one is there. So I'm wondering if this second Analytics code is effecting our traffic and reporting, even if the code is currently non-functional? and if it is, how could I go about removing it if it's not currently in our HTML? Thanks!
Reporting & Analytics | | PlanetDISH0 -
PDFs and indexing
Hello and good morning. I work for a paint manufacturing company in the UK on their seo campaigns across a couple of websites, this is my question. as paint and chemicals require data and tech sheets by law, available to be downloadable for said product, should these be included in the sitemap, we auto generate our sitemaps which they include these files, with low priorities and never change in terms of name etc. they basically have a name of say 092847.pdf for example which cannot be changed, but from an seo view this doesn't mean a thing? so theres my question should they be included and would they carry any value?
Reporting & Analytics | | TeamacPaints0 -
Google Analytics
Good Morning, I am trying to understand 2 issues in Google Analytics. 1. When look at : Traffic Source --> SEO --> Quesrios - i see the impressions column and its always a whole number 1550, 500, 5500, etc.. I never saw (for example) 702, 313, etc... impressions Can anyone explain why and how does it work? 2. In the same report i see my AVG. position for each query, the question is how come i have AVG, position of (290, 230, 190) for some of the queries and still i get clicks on these queries. My guess is that from time to time these queries have better position and the clicks are from these time. Do you familiar with a way to the the distribution of a specific query over time? for example: 1.3.2013 avg position = 4 2.3.2013 avg position = 7 3.3.2013 avg position = 2 4.3.2013 avg position = 8 etc... 3. This report say its for: "Top 1,000 daily queries" - What does it mean? Thank you and sorry for this long question SEOwiseUs
Reporting & Analytics | | iivgi0 -
"not selected" is gone from Google Webmaster Tools Index Status Advanced
Just noticed today that the "not selected" has been removed from the Index status, Advanced section of Google Webmaster Tools. Anyone know why. I've used this metric to determine how or why Google was not selecting pages, particularly to gauge canonical's, 301's and duplicate content. It will be missed if gone for good.
Reporting & Analytics | | tdawson090 -
Does Google Analytics use your data against you?
I couldn't find this question answered anywhere in Q&A, so I apologize if it's a duplicate of another post. I heard, about a year ago on either Web Pro World, or Warrior Forum that Google uses your visitor data in your Google Analytic account to rank your site. Someone said that when they took out the Google code, their site went from the third to the first page within 48 hours. That was then verified over the next couple of weeks by others. Their thought was that regardless of the optimized page and incoming link, if the site wasn't getting visitors, then it would be penalized. Since Google has the data, they would be able to use it. I then started using another, paid, solution - getclicky.com. While I like clicky, there is some info Google has that clicky doesn't, everyone integrates with Google analytics - like SEOmoz, and I'm paying a monthly fee. Now that I'm a part of a community of experts, what do you think? Have you noticed Google ranking you based on your analytics data? Has anyone experienced this, or heard about it before? Because I'd like to go back to using Google analytics. Thanks!
Reporting & Analytics | | DallasBonsai0