Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
How to safely exclude search result pages from Google's index?
-
Hello everyone,
I'm wondering what's the best way to prevent/block search result pages from being indexed by Google. The way search works on my site is that search form generates URLs like:
/index.php?blah-blah-search-results-blahI wanted to block everything of that sort, but how do I do it without blocking /index.php ?
Thanks in advance and have a great day everyone!
-
Hi Louise,
If you can ID the parameters, you can also look at blocking these in Webmaster Tools. This page explains more. As with any blocking of URLs, of course, proceed with caution.
-
I agree that can be effective. The reason I suggested the robots.txt is because Louise mentioned "blocking and preventing" as an objective. Robots.txt are particularly useful in the example where results from a search bar or something of that nature is involved. A NOINDEX, FOLLOW will not prevent bots from getting tired and dizzy, whereas the robots.txt can "block and prevent" bots from crawling certain parameters.
With all of that said, I think it is important to understand whether you need the bots to crawl and not index (in which case Spencer's answer is correct), or if you need to prevent bots from crawling the parameters altogether.
Hope that is more clear

-
I'm not sure that robots.txt is effective when url parameters are involved.
I would just add a meta robots tag to the head section of the search results template:
-
If you are able to identify a url parameter, you may excluded them using robots.txt. Here is a great resource on Robots.txt - http://moz.com/learn/seo/robotstxt
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Strange landing page in Google Analytics
Hello MOZ Community, The website in question is https://x-y.com/ When i looked at the landing pages report in GA , x-y.com is appended at the end of every URL like this. https://x-y.com/x-y.com When i open the above URL in GA interface, it shows page not found. This is obvious as there is no such URL.
Reporting & Analytics | | Johnroger
The metrics like sessions, Users, Bounce rate all look good. In the property settings, The default URL is written like this http:// cell-gate.com (Please note that s is missing in property settings). But how is traffic tracked correctly How do i solve this problem. What settings should we change to make the landing pages report look ok Thanks0 -
How to track google auto search suggestion click?
Hello Guys, In google.co.uk when I search SEL and google gives me option of different different sites and when I click on any one site then that click tracking I need. I have attached the screenshot to understand easily. Is it possible to track such things or possible via server logs etc? TV99h
Reporting & Analytics | | micey1231 -
Conflicting average position data from Google Search Console?
I'm looking at Google Search Console data in Google Analytics, specifically Average Position as given in the Landing Page report, and the same metric broken out by mobile and desktop in the Devices report. In the Landing Page report, I see an aggregated average position that's much higher/worse than an actual average of what is reported for mobile, desktop and tablet traffic under the Device reporting. For example: Mobile: 5 Desktop: 5 Tablet: 5 So the average still should be roughly 5, right? Why would the Landing Page then show an aggregate Average Position of 8? I wouldn't expect to see a precisely same average given that different device types have different proportions that could render differently when the buckets are combined, but this is a huge swing. In fact, the aggregate Average Position as given in the top level Devices report is closer to 5 than to the 8 shown in the Landing Pages report. (These aren't actual numbers, but are illustrative of what I'm seeing, by the way.) Unless I'm missing some vital difference in the way that Average Position is reporting for the Landing Page report versus the Device reports, it doesn't seem like this should be possible. What am I missing?
Reporting & Analytics | | BradsDeals0 -
"index.htm" for all url's in google analytics
I don't have this issue with other wordpress websites, only this one website, and I don't know what's causing the issue: Google Analytics is adding an "index.htm" to every single page on the website. So it is tracking the pages, I see no errors - is it tracking the right page? When I click on the page link in a report, I naturally go to a "404 page not found" since the website address isn't "www.example.com/rewards/index.htm" - but instead the actual address would be:
Reporting & Analytics | | cceebar
"www.example.com/rewards/". I have navigated to View Settings in GA to insure "default page" is empty. Although adding anything else to this field does not effect the page url in analytics reports either. Could it be htaccess file - or a plugin effecting the htaccess file?_Cindy0 -
Why google stubbornly keeps indexing my http urls instead of the https ones?
I moved everything to https in November, but there are plenty of pages which are still indexed by google as http instead of https, and I am wondering why. Example: http://www.gomme-auto.it/pneumatici/barum correctly redirect permanently to https://www.gomme-auto.it/pneumatici/barum Nevertheless if you search for pneumatici barum: https://www.google.it/search?q=pneumatici+barum&oq=pneumatici+barum The third organic result listed is still http. Since we moved to https google crawler visited that page tens of time, last one two days ago. But doesn't seems to care to update the protocol in google index. Anyone knows why? My concern is when I use API like semrush and ahrefs I have to do it twice to try both http and https, for a total of around 65k urls I waste a lot of my quota.
Reporting & Analytics | | max.favilli0 -
Switch to www from non www preference negatively hit # pages indexed
I have a client whose site did not use the www preference but rather the non www form of the url. We were having trouble seeing some high quality inlinks and I wondered if the redirect to the non www site from the links was making it hard for us to track. After some reading, it seemed we should be using the www version for better SEO anyway so I made a change on Monday but had a major hit to the number of pages being indexed by Thursday. Freaking me out mildly. What are people's thoughts? I think I should roll back the www change asap - or am I jumping the gun?
Reporting & Analytics | | BrigitteMN0 -
Google Analytics Organic Search Keywords Suddenly Displaying FulL Urls
In my Google Analytics, the top keywords for Organic Search are suddenyl displaying full URLs. For example, now the third and fourth keywords are http://www.domain.com/highly-specific-URL. These have all started recently around the same day, July 12th. I've checked back, and we've made no internal changes to the site around that time that could affect this. Any thoughts on this? Thanks! P.S. It might be related to rich snippets, but I cannot tell at this point.
Reporting & Analytics | | 10SL0 -
Weird info from google analytics?
Hi Could anyone explain what these visits are in Google Analytics? Under traffic sources and organic I am seeing lots of entries with data like below. Any ideas what kind of traffic this is? Is it a bot and if so what is their purpose of it and is it recommended that you block it? Pages/Visit 1.00 Avg. Time on Site 00:00:00 % New Visits : 100% Bounce Rate: 100.00% Many Thanks
Reporting & Analytics | | ocelot0