How to safely exclude search result pages from Google's index?
-
Hello everyone,
I'm wondering what's the best way to prevent/block search result pages from being indexed by Google. The way search works on my site is that search form generates URLs like:
/index.php?blah-blah-search-results-blahI wanted to block everything of that sort, but how do I do it without blocking /index.php ?
Thanks in advance and have a great day everyone!
-
Hi Louise,
If you can ID the parameters, you can also look at blocking these in Webmaster Tools. This page explains more. As with any blocking of URLs, of course, proceed with caution.
-
I agree that can be effective. The reason I suggested the robots.txt is because Louise mentioned "blocking and preventing" as an objective. Robots.txt are particularly useful in the example where results from a search bar or something of that nature is involved. A NOINDEX, FOLLOW will not prevent bots from getting tired and dizzy, whereas the robots.txt can "block and prevent" bots from crawling certain parameters.
With all of that said, I think it is important to understand whether you need the bots to crawl and not index (in which case Spencer's answer is correct), or if you need to prevent bots from crawling the parameters altogether.
Hope that is more clear
-
I'm not sure that robots.txt is effective when url parameters are involved.
I would just add a meta robots tag to the head section of the search results template:
-
If you are able to identify a url parameter, you may excluded them using robots.txt. Here is a great resource on Robots.txt - http://moz.com/learn/seo/robotstxt
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Can Anyone help me with Google search console soft 404 error?
Hello everyone, I just build one site on WordPress and submitted it to the Google search console along with the sitemap. Some URLs got indexed but one of my URL is showing error in Search console. My post https://hotpass.site/create-subdomain-godaddy/ is showing soft error 404 webpage not available in Google search console. I have rechecked the site and seo settings. Nothing is there. I have even checked it with LIVE URL test. But still Search console is not accepting this URL. Can anyone help me with this? Help will be appriciated.
Reporting & Analytics | | Pauline210 -
More 'New users' than 'Users' in Google Analytics
Hey Mozzers, I'm seeing more 'New users' than 'Users' in Google Analytics (see screenshot). This is over a 3 year time period. Any idea why new users would be greater than users? Thanks! 6lYmU0j
Reporting & Analytics | | RWesley0 -
Webmaster Tools Pages and Queries
In Google Webmaster Tools, under Search Analytics, what is the difference between positions in queries and positions in pages?
Reporting & Analytics | | CostumeD0 -
Google Analytics View Filters
Using the same GA property, I would like to set up three filtered views: 1. Tracking across one subdomain and one primary domain (example: shop.example.com & example.com) 2. Track only primary domain (example.com) 3. Track only subdomain (shop.example.com) Can this be achieved by using view filters? If so, how do they need to be set? Also, according to this article: https://moz.com/blog/cross-domain-subdomain-tracking-in-google-analytics, with cross domain tracking, I need to ignore self-referrals, which can only be done at the property level. If set up to ignore example.com referrals, will this cause problems with filter 2 and 3?
Reporting & Analytics | | Evan340 -
Why only a few pages of my website are being indexed by google
Our website www.navisyachts.com has in its sitemap over 3000 pages of information, and this is all unique content written by our team. Now Google Webmaster central shows only 100 urls indexed from 3500 submitted. Can you help me understand why and how I can fix this issue? The website has 4 years old, is a Joomla 3.3 up to date. It has part of the content in the Joomla core content systems and part in K2. Thank you. Pablo
Reporting & Analytics | | FWC_SEO0 -
I want to take down some pages, how do I inform google?
Hey Guys, I'm hoping someone can help - I'm in the midst of a site re-design - whilst one of our biggest reasons for the redesign was to create more space to write valuable content and unique content I have been reading other posts on moz about content auditing. I have come across a few articles in my own blog that are from 250 - 300 words to which the articles seem similar and the traffic is low. I'm wondering while I'm left to consolidate these articles and create a fresh article for each entry that is more in-depth, when I consolidate or delete these pages - do I need to inform google these pages have now been deleted? If so using Wordpress what is the best way to do this? Cheers and would appreciate some advise Thanks
Reporting & Analytics | | edward-may0 -
Pro's & Con's of Wordpress Categorys & Tags
Good Afternoon! I touched on this question a while back in another post specifically regarding a plethora of duplicate pages that I was finding due to inappropriate tagging in wordpress. As I am going through our website, I am starting to notice it happening again with categories as well. I am including some pictures where you can see the URL structures and titles etc of how everything is laid out. I would like to clarify that I was not the one who did any of this Is it wrong/bad to cross categorize? What I mean by that is put something in more than one category? Would there be any drawback to converting any of these into subcategories? Would that even do anything? Does having two pages that are named the same thing, hurt you? It would seem to me that Google wouldn't like that. I have recently come into the field of thought that Google is getting more and more human, and If it makes a human uncomfortable/confused it will make Google confused. In my pictures you can see we clearly have numerous hard copies of the same thing, not just duplicate elements created by wordpress, that is a separate issue. I personally want to change all of the titles and make everything as different and individual as possible, but i also could be very wrong in my desire to do that. Any thoughts are appreciated! eY4iX2N N3AVqss JZpU7Rq
Reporting & Analytics | | HashtagHustler0 -
301 redirect visible on Google.
Hello, This is a strange one for me. We have 301 redirected our .com domain to the .co.uk domain. The strange thing is that if you gogole the .com domain (theloanengine.com) there is a result with the website description. If you click on the result you are redirected tot he website homepage. One other thing: I've discovered this 301 issue because Google Analytics started to show me a few days ago referrals from the .com domain. I don't know if these things are connected. Cornel
Reporting & Analytics | | Cornel_Ilea0