How to safely exclude search result pages from Google's index?
-
Hello everyone,
I'm wondering what's the best way to prevent/block search result pages from being indexed by Google. The way search works on my site is that search form generates URLs like:
/index.php?blah-blah-search-results-blahI wanted to block everything of that sort, but how do I do it without blocking /index.php ?
Thanks in advance and have a great day everyone!
-
Hi Louise,
If you can ID the parameters, you can also look at blocking these in Webmaster Tools. This page explains more. As with any blocking of URLs, of course, proceed with caution.
-
I agree that can be effective. The reason I suggested the robots.txt is because Louise mentioned "blocking and preventing" as an objective. Robots.txt are particularly useful in the example where results from a search bar or something of that nature is involved. A NOINDEX, FOLLOW will not prevent bots from getting tired and dizzy, whereas the robots.txt can "block and prevent" bots from crawling certain parameters.
With all of that said, I think it is important to understand whether you need the bots to crawl and not index (in which case Spencer's answer is correct), or if you need to prevent bots from crawling the parameters altogether.
Hope that is more clear
-
I'm not sure that robots.txt is effective when url parameters are involved.
I would just add a meta robots tag to the head section of the search results template:
-
If you are able to identify a url parameter, you may excluded them using robots.txt. Here is a great resource on Robots.txt - http://moz.com/learn/seo/robotstxt
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Strange - Search Console page indexing "../Detected" as 404
Anyone seen this lately? All of a sudden Google Search Console is insisting in Page indexing that there is a 404 for a page that has never existed on our client's site: https://........com.au/Detected We've noticed this across a number of sites, precisely in this way with a capitalised "/Detected" To me it looks like something spammy is being submitted to the SERPs (somehow) and Google is trying to index that and then getting a 404. Naturally MOZ isn't picking it up, cause the page simply never existed - it's just happening in Search Console 2afc7e35-71e4-4e25-80a3-690bf10776a7.png It comes and it goes in the 404 alerts in Console and is really annoying. I reckon it started happening late 2022.
Reporting & Analytics | | DanielDL0 -
Pandora in Google Analytics
Hi Mozers, in Google Analytics our ads are registering a 10 second page duration and a 90% bounce rate, which is pretty atrocious. My questions are: Organic page duration is 3:15 minutes, and Social + Referral are similar. Yet Pandora is bringing the average time on site way down, to less than a minute. Will those paid ad metrics be factored to the User Behavior and therefore negative affect our overall rankings? In other words does Google now see our page duration as less than a minute? The Pandora rep says that Google Analytics and Pandora have "systemic issues" such that GA is very inaccurate with Pandora metrics. I find that difficult to believe. Any insights here? Thanks all! Jane
Reporting & Analytics | | CalamityJane770 -
Google Analytics Content Experiments - Experiment Conversions and Goal Figures Don't Match
Hi, I set up a new content experiment 6 days ago, the experiment says there have been 2 conversions but the goal associated with it says 5. The experiment is set to target 100% of traffic, distributed evenly among the variations, the goal is a destination URL goal. I've doubled checked the goal set up and everything seems fine. How can the content experiment report a different figure to the goal associated with it? Has anyone else noticed the same problem? Is this a bug? Is there a workaround available? Or is there a setting I need to be aware of when creating content experiments to prevent this from happening? I need to know I can trust the results the content experiments provide.
Reporting & Analytics | | UNIT40 -
How long after a redirect should we see results?
In mid September we had to so some redirects from old URL's to new URL's. We haven't seen a significant jump in traffic so we are wondering how long this might take before we see a change. We are the brand so we should be getting some sort of authority from Google, yes? Any tips that can be offered would be grateful as we are stumped.
Reporting & Analytics | | K2_Sports0 -
301 redirect visible on Google.
Hello, This is a strange one for me. We have 301 redirected our .com domain to the .co.uk domain. The strange thing is that if you gogole the .com domain (theloanengine.com) there is a result with the website description. If you click on the result you are redirected tot he website homepage. One other thing: I've discovered this 301 issue because Google Analytics started to show me a few days ago referrals from the .com domain. I don't know if these things are connected. Cornel
Reporting & Analytics | | Cornel_Ilea0 -
Drop in google referral traffic
Hi guys, As we know, GA shows google as traffic source in two ways: google / organic for organic searches and google.TLD / referral for everything else: google groups, base.google.com, static pages, google reader, google image search, google search appliance/mini. What we noticed is that around Oct 20th there's a huge drop of google.TLD / referral traffic to our site. Do you experience something similar? I couldn't find anything Google-related that happened around this specific date. We use GSA for our site search and I'm wondering if this could be the reason - maybe someone from our development team made changes to GSA settings that affected this traffic source. Looking forward to hearing from you! Thanks.
Reporting & Analytics | | lgrozeva0 -
Google Analytics | REAL TIME
So I noticed today that there is now Real Time Data: http://analytics.blogspot.com/2011/09/whats-happening-on-your-site-right-now.html and I cannot figure out how to access this.
Reporting & Analytics | | joseph.chambers1 -
Will Google start trimming 'stale' sites rank?
With the recent focus on Google to reduce rank of farms and low value sites, I am interested to get SEO view on if you think Google will start devaluing stale sites. I do find it a bit frustrating that in the top 5 for my main key phrase, there is one site that has NO content just an error and another blog that has not updated content in 2 years. How can blogs that do not blog be considered high enough value by Google to rank in the top 5? How can sites that just return 404 or 500 for ALL their pages be even considered a site let alone rank 2nd. I am interested so see others experiences and thoughts on 'user experience' clean ups by Google and why these types of sites get missed?
Reporting & Analytics | | oznappies0