How to safely exclude search result pages from Google's index?
-
Hello everyone,
I'm wondering what's the best way to prevent/block search result pages from being indexed by Google. The way search works on my site is that search form generates URLs like:
/index.php?blah-blah-search-results-blahI wanted to block everything of that sort, but how do I do it without blocking /index.php ?
Thanks in advance and have a great day everyone!
-
Hi Louise,
If you can ID the parameters, you can also look at blocking these in Webmaster Tools. This page explains more. As with any blocking of URLs, of course, proceed with caution.
-
I agree that can be effective. The reason I suggested the robots.txt is because Louise mentioned "blocking and preventing" as an objective. Robots.txt are particularly useful in the example where results from a search bar or something of that nature is involved. A NOINDEX, FOLLOW will not prevent bots from getting tired and dizzy, whereas the robots.txt can "block and prevent" bots from crawling certain parameters.
With all of that said, I think it is important to understand whether you need the bots to crawl and not index (in which case Spencer's answer is correct), or if you need to prevent bots from crawling the parameters altogether.
Hope that is more clear
-
I'm not sure that robots.txt is effective when url parameters are involved.
I would just add a meta robots tag to the head section of the search results template:
-
If you are able to identify a url parameter, you may excluded them using robots.txt. Here is a great resource on Robots.txt - http://moz.com/learn/seo/robotstxt
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Phantom search results showing up in analytics
Hi, I'm getting these weird phantom search results showing up in my analytics account which is skewing my results. I have a wordpress site, and I do have some custom search results pages, but analytics is telling me these hits are coming from google- which I doubt. The end of the query string (&cat=plus-5-results) is not coming from a link created within my site. Anyone have any ideas? Failing that, would somebody be kind enough to explain how I can ignore these hits, but retain other, valid search result requests? Thanks reading! Will Screenshot%202019-05-16%20at%2022.32.10.jpg
Reporting & Analytics | | madegood0 -
Rel=Canonical vs. No Index
Ok, this is a long winded one. We're going to spell out what we've seen, then give a few questions to answer below, so please bear with us! We have websites with products listed on them and are looking for guidance on whether to use rel=canonical or some version of No Index for our filtered product listing pages. We work with a couple different website providers and have seen both strategies used. Right now, one of our web providers uses No Index, No Follow tags and Moz alerted us to the high frequency of these tags. We want to make sure our internal linking structure is sound and we are worried that blocking these filtered pages is keeping our product pages from being as relevant as they could be. We've seen recommendations to use No Index, Follow tags instead, but our other web provider uses a different method altogether. Another vendor uses a rel=canonical strategy which we've also seen when researching Nike and Amazon's sites. Because these are industry leading sites, we're wondering if we should get rid of the No Index tags completely and switch to the canonical strategy for our internal links. On that same provider's sites, we've found rel=canonical tags used after the first page of our product listings, and we've seen recommendations to use rel=prev and rel=next instead. With all that being said, we have three questions: 1)Which strategy (rel=canonical vs. No Index) do you recommend as being optimal for website crawlers and boosting our site relevance? 2)If we should be using some version of No Index, should we use Follow or No Follow? 2)Depending on the product, we have multiple pages of products for each category. Should we use rel=prev & rel=next instead of rel=canonical among the pages after page one? Thanks in advance!
Reporting & Analytics | | Leithmarketing0 -
Google Analytics Average Position
I'm looking at Google Analytics -> Acquisition -> Search Engine Optimization -> Queries reports. I'm looking at keywords and the average position. What Google reports and what I see in a Google incognito search is different (usually my search is much lower). For example, for one search term, Google reports 5.8 average position and every time I search it is 8. My local result is 4. Anyone know why this is? I'm wondering if Google is averaging the Local results into number?
Reporting & Analytics | | CalicoKitty20000 -
How can you add a rel canonical tag if you haven't created the wrong pages?
For one of our white paper campaigns we are getting multiple URL's some how but we only have one version of the page. So do I put the rel canonical tag on that one single page? Will that fix the other url's from being indexed? I'm assuming people are typing in the urls with underscores and capital or non-capital letters and it's showing up that way in analytics. Thanks!
Reporting & Analytics | | Sika220 -
Google Analytics Code
We have a quick question about our Google Analytics code: We recently updated to a re-marketing Analytics code, and some of our traffic numbers seem to be off by a bit. I used the Google Tag Assistant Chrome Extension today, and noticed that it's finding our old Analytics code on our pages, but it's coming up as an error due to "no HTTP Response". I am attempting to remove this code from the website, but it is nowhere to be found in the HTML coding. Only the current one is there. So I'm wondering if this second Analytics code is effecting our traffic and reporting, even if the code is currently non-functional? and if it is, how could I go about removing it if it's not currently in our HTML? Thanks!
Reporting & Analytics | | PlanetDISH0 -
Traffic Down for Most Referrers - not just Google
Our traffic has taken a severe hit, over the past 3 weeks - down about 60%, which I had assumed was caused by the Penguin update. However on closer inspection of our analytics, our traffic is is down by between 30 and 50% for nearly all our referrers - including Bing and other search engines, referring sites, and even direct traffic! Google provides the vast majority of our traffic, so in terms of the absolute visitor count, the drop in Google traffic has the biggest effect - by some distance. But the fact that traffic is down by similar percentages suggest that Penguin isn't the cause of our troubles at all. We sell garden products in the UK, and it's just coming into peak season. Last year, May was one of our top months, External conditions - such as the very wet weather over the past month, economic gloom and doom - don't begin to explain this sudden and dramatic drop in traffic. We are very perplexed. If anyone has any bright ideas, I'd be interested to hear them. Ben
Reporting & Analytics | | atticus70 -
Un-link Google Analytics
I have set the wrong account/password details for one of my campaigns. How do I 'step back' and choose the correct settings please? Thanks Ian
Reporting & Analytics | | driansmith0 -
Google +1 and ranking effect .
Anyone have any experience as to what effect google +1 has on ranking? Trying to figure out what impact it might have. Ecommerce site in norway selling childrens clothing. Atm we are ranking 25th for main targetted word childrens clothing (barneklær in norwegian). on google.no this has a difficulty of 34. None of main competitors have any amount of google +1's on their site. Lets say if I in some way could manage to get 100 ppl to push +1 button. Would this affect our ranking at all ? Might be impossible to answer but if anyone has thoughts about it I am interested. Thanks
Reporting & Analytics | | danlae0
