Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
How to safely exclude search result pages from Google's index?
-
Hello everyone,
I'm wondering what's the best way to prevent/block search result pages from being indexed by Google. The way search works on my site is that search form generates URLs like:
/index.php?blah-blah-search-results-blahI wanted to block everything of that sort, but how do I do it without blocking /index.php ?
Thanks in advance and have a great day everyone!
-
Hi Louise,
If you can ID the parameters, you can also look at blocking these in Webmaster Tools. This page explains more. As with any blocking of URLs, of course, proceed with caution.
-
I agree that can be effective. The reason I suggested the robots.txt is because Louise mentioned "blocking and preventing" as an objective. Robots.txt are particularly useful in the example where results from a search bar or something of that nature is involved. A NOINDEX, FOLLOW will not prevent bots from getting tired and dizzy, whereas the robots.txt can "block and prevent" bots from crawling certain parameters.
With all of that said, I think it is important to understand whether you need the bots to crawl and not index (in which case Spencer's answer is correct), or if you need to prevent bots from crawling the parameters altogether.
Hope that is more clear
-
I'm not sure that robots.txt is effective when url parameters are involved.
I would just add a meta robots tag to the head section of the search results template:
-
If you are able to identify a url parameter, you may excluded them using robots.txt. Here is a great resource on Robots.txt - http://moz.com/learn/seo/robotstxt
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Fire a tag when element is loaded on page (Google Tag Manager)
I'm using an Element Visibility trigger to track a value that appears on a page. However, I want to track this value even when the user doesn't scroll to the area of the page where the element is (i.e. when the page is loaded, and the value is displayed below the fold, but the user doesn't scroll down there). Is there a way of doing this
Reporting & Analytics | | RWesley0 -
Moz reports way fewer backlinks than google search?
My site is only 11 months old but has steadily (if not slowly) been gaining backlinks. My question, is why Moz shows me at 303 backlinks and Google search console is showing at 1,237? I am more than a little suspicious that this could highlight the reason Moz shows such an unfavorable DA ranking for our site at a DA12. Other competitors that rank for similar keywords to mine are DA 42, DA 65, DA 73, etc. If the largest ranking factor is links, and they have mine reported incorrectly - is this the issue with DA as it relates to sites like mine? Any answer from someone who has experienced similar, or has a definitive answer is more than welcome to chime in! Thanks, Kevin
Reporting & Analytics | | kvncrll0 -
Redirecting all URLs appended with index.htm or index.html
It has come to my attention with one of my clients (WordPress website) that for some time they have within their Landing Page report (of GA - Google Analytics) URLs that should all be pointing to the one page, example: domain.com/about-us, also has a listing in GA as domain.com/about-us/index.htm Is this some kind of indication of a subdirectory issue? Has anyone had experience with this in such wordpress plugins as Yoast SEO, or other SEO plugin? My thoughts here are to simply redirect any of these non-existent files with a redirect in .htaccess - but what I'm using isn't working. I will insert the redirect here - - and any help would be greatly appreciated. RewriteEngine onRewriteCond %{THE_REQUEST} ^./index.html?
Reporting & Analytics | | cceebar
RewriteRule ^(.)index.html?$ http://www.dupontservicecenter.com/$1 [R=301,L] and this rewrite doesn't work: RewriteEngine on
RewriteRule ^(.+).htm$ http://dupontservicecenter.com/$1.php [R,NC] _Cindy0 -
Google Analytics - Organic Search Traffic & Queries -What caused the huge difference?
Our website traffic dropped a little bit during the last month, but it's getting better now, almost the same with previous period. But our conversion rate dropped by 50% for the last three weeks. What could cause this huge drop in conversion rate? In Google Analytics, I compared the Organic Search Traffic with previous period, the result is similar. But the Search Engine Optimization ->Queries shows that the clicks for last month is almost zero. What could be the cause of this huge differnce? e9sJNwD.png k4M8Fa5.png
Reporting & Analytics | | joony0 -
Getting google impressions for a site not in the index...
Hi all Wondering if i could pick the brains of those wise than myself... my client has an https website with tons of pages indexed and all ranking well, however somehow they managed to also set their server up so that non https versions of the pages were getting indexed and thus we had the same page indexed twice in the engine but on slightly different urls (it uses a cms so all the internal links are relative too). The non https is mainly used as a dev testing environment. Upon seeing this we did a google remove request in WMT, and added noindex in the robots and that saw the index pages drop over night. See image 1. However, the site still appears to getting return for a couple of 100 searches a day! The main site gets about 25,000 impressions so it's way down but i'm puzzled as to how a site which has been blocked can appear for that many searches and if we are still liable for duplicate content issues. Any thoughts are most welcome. Sorry, I am unable to share the site name i'm afraid. Client is very strict on this. Thanks, Carl image1.png
Reporting & Analytics | | carl_daedricdigital0 -
Google Ad referral
I was wondering if someone could decode the jumble of a referral - this is supposedly the referal that led to a click through to my site via a product listing ad. I am trying to figure out how www.nextag.com comes in to the picture as we do not have refurbexperts even listed there? Thanks to anyone who tries/does work it out. http://www.googleadservices.com/pagead/aclk?sa=L&ai=CGXud6DmDU_qeL5THygHpuICwCaTZwMYD_Nvvv0bEwMS50wEIBhAEIOn5-gEoBVCl7P7f-v____8BYMnu8omYpPQSoAHAhIv9A8gBB8gDG6oEJ0_QwcNc5zNun_d7S5KNcMT6uPjjH_mMDkKFFgBCQ6aKICRPJVVa7MAFBYgGAaAGJoAHqPv0ApAHAeASupqdo-ypit0m&ohost=www.google.com&cid=5GhZEzUCSC6x9n2wxOdz3-mrAfSUkvHKPN3wD5yLInnlNil_&sig=AOD64_1D1z1JPYbFP0UnUglJVOfvd25RfA&adurl=http://refurbexperts.com/product/527/HP-LaserJet-P2015-Laser-Printer-RECONDITIONED%3Futm_source%3Dproductlistingads%26utm_medium%3Dadwords%26utm_campaign%3Dadwords&ctype=5&nb=0&res_url=http%3A%2F%2Fwww.nextag.com%2Fhp-p2015-laserjet%2Fproducts-html%3Fnxtg%3D116d0a1c0504-9FFEB16DE52A7E2A&rurl=http%3A%2F%2Fwww.nextag.com%2Fgoto.jsp%3Fp%3D3652%26search%3Dhp%2520p2015%2520laserjet%26t%3Dag%253D1384181795%26crid%3D48271786%26gg_aid%3D20169721025%26gg_site%3D%26gclid%3DCjgKEAjwzIucBRDzjIz9qMOB3TASJABBIwL1LHK7GcAPS6yHGpd9Kq3wsZrcPORAWD8QCWivr4W75PD_BwE&nm=11&nx=43&ny=12&is=700x181&clkt=187
Reporting & Analytics | | henya0 -
Google Analytics - Next Page Path is the Same URL?
Hey Everyone, I have a Google analytics question. I'm looking through a client's site and when I look at the next page path, I get the same URL as the next path. For example, on the homepage, the next page path I get is the homepage again? This happens for all URL's, is this an implementation error? Is there a way to fix this? Thanks!
Reporting & Analytics | | EvansHunt0 -
Google Analytics and DNS change
Our new alumni application is going be tested at domain uva.imodules.com . We are going to collect traffic data with a Google analytics account number UA-884652-XX. So going to uva.imodules.com/myPage.html would send its data to Google Analytics with that account number. Then when it is ready for production we are going to just change the domain name of the application and switch the DNS over to dardencommunity.darden.virginia.edu . So going to dardencommunity.darden.virginia.edu /myPage.html would send its data to Google Analtics with that SAME account number. Aside from having the testing domain data in the same profile are there any other issues/problems we may run into?
Reporting & Analytics | | Darden0