Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
How to safely exclude search result pages from Google's index?
-
Hello everyone,
I'm wondering what's the best way to prevent/block search result pages from being indexed by Google. The way search works on my site is that search form generates URLs like:
/index.php?blah-blah-search-results-blahI wanted to block everything of that sort, but how do I do it without blocking /index.php ?
Thanks in advance and have a great day everyone!
-
Hi Louise,
If you can ID the parameters, you can also look at blocking these in Webmaster Tools. This page explains more. As with any blocking of URLs, of course, proceed with caution.
-
I agree that can be effective. The reason I suggested the robots.txt is because Louise mentioned "blocking and preventing" as an objective. Robots.txt are particularly useful in the example where results from a search bar or something of that nature is involved. A NOINDEX, FOLLOW will not prevent bots from getting tired and dizzy, whereas the robots.txt can "block and prevent" bots from crawling certain parameters.
With all of that said, I think it is important to understand whether you need the bots to crawl and not index (in which case Spencer's answer is correct), or if you need to prevent bots from crawling the parameters altogether.
Hope that is more clear
-
I'm not sure that robots.txt is effective when url parameters are involved.
I would just add a meta robots tag to the head section of the search results template:
-
If you are able to identify a url parameter, you may excluded them using robots.txt. Here is a great resource on Robots.txt - http://moz.com/learn/seo/robotstxt
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Website excluded from indexing, google-selected canonical: N/A
The google search console revealed to me that none of my pages was indexed, all pages are listed in the 'excluded' section "duplicate, google chose different canonical than user".
Reporting & Analytics | | Fibigrus
But in the URL-inspection tab it shows me google-selected canonical: N/A Indexing and crawling is both allowed. Don't know how to get my pages to be indexed correctly. (by the way, they do NOT exist in other languages, so that can't be a reason why google might think they are a duplicate. There's definitively no other version of those pages available)0 -
How to exclude traffic for a specific mobile device in Google Analytics view?
Hi, Need help on how to exclude traffic for a specific mobile device in Google Analytics view. I have been searching and the only information available is excluding IP address of internal traffic. Is there any way to exclude traffic through a mobile MAC address?
Reporting & Analytics | | Khadija_K0 -
Migrated website but Google Analytics still displays old URL's and none new?!
I migrated a website from a .aspx to a .php and hence had to 301 all the old urls to the new php ones. It's been months after and I'm not seeing any of the php pages showing results but I'm still getting results from the old .aspx pages. Has any one had any experience with this issue or knows what to do? Many thanks,
Reporting & Analytics | | CoGri0 -
Referral Traffic from Google
Hello, I have a question about my company's new website. I've worked in SEO and studied Google Analytics results for a few years now but have never really come across something like this. I started in this position in January of this year and when I started breaking down the traffic sources in Google Analytics, I noticed most of the traffic was coming from Google.com as a referral source. I had never seen Google.com as a referral source before so I looked into options for what it could be. It was not a paid ad and our organic traffic was coming through in Analytics, Before I could get any further, our new website was launched (we switched CRM's to WordPress) and the referral traffic from google went from 2,966 in January of 2015 to 22 in February 2015. for more comparison, in February of 2014, the referral traffic from Google was 2,496. I expected a drop when we switched CRM's but we correctly re-directed all pages and created a new sitemap and our organic traffic is up since the switch (not enough to cover drop in referral). I thought at first this had to do with our Google sellers account being de-activated when we made the switch, but I quickly fixed this over a month ago and no change. I'm wondering if anyone has ever seen Google.com come through as a referral source in Google Analytics and if they we're able to figure out what it actually was. This would be a great help! Thank you, Alex
Reporting & Analytics | | RASEO1 -
Why is Google Analytics showing index.php after every page URL?
Hi, My client's site has GA tracking code gathering correct data on the site, but the pages are listed in GA as having /index.php at the end of every URL, although this does not appear when you visit the site pages. Even if there is a redirect happening for site visitors, shouldn't GA be showing the pages as their redirect destination, i.e. the URL that visitors actually see? Could this discrepancy be adversely affecting my search performance? Example page: http://freshstarttax.com/innocent-spouse/ shows up in GA as http://freshstarttax.com/innocent-spouse/index.php thanks
Reporting & Analytics | | JMagary0 -
Setting up Google Analytics for Subsites
I currently have one main .com site and am planning on launching geo-location subsites .co.uk, .com.au, .ru, etc... Traffic will flow between both sites and some of the content on the subsites will be duplicate and therefore include a canonical tag to the main site. I want to set up GA to capture who is going to the subsites and vice versa and correctly capture crossover traffic. Any advice on implementing advanced analytics directly (or links to sources that will direct me the right direction for this project)
Reporting & Analytics | | theLotter0 -
Weird info from google analytics?
Hi Could anyone explain what these visits are in Google Analytics? Under traffic sources and organic I am seeing lots of entries with data like below. Any ideas what kind of traffic this is? Is it a bot and if so what is their purpose of it and is it recommended that you block it? Pages/Visit 1.00 Avg. Time on Site 00:00:00 % New Visits : 100% Bounce Rate: 100.00% Many Thanks
Reporting & Analytics | | ocelot0 -
Exclude Traffic from India
i would like to exclude all traffic coming from India using Advanced Segment in Google Analytics. How do i go about it ? Will it be applied to future traffic also ?
Reporting & Analytics | | seoug_20050