On-site Search - Revisited (again, *zZz*)
-
Howdy Moz fans!
Okay so there's a mountain of information out there on the webernet about internal search results... but i'm finding some contradiction and a lot of pre-2014 stuff. Id like to hear some 2016 opinion and specifically around a couple of thoughts of my own, as well as some i've deduced from other sources. For clarity, I work on a large retail site with over 4 million products (product pages), and my predicament is thus - I want Google to be able to find and rank my product pages. Yes, I can link to a number of the best ones by creating well planned links via categorisation, silos, efficient menus etc (done), but can I utilise site search for this purpose?
-
It was my understanding that Google bots don't/can't/won't use a search function... how could it? It's like expeciting it to find your members only area, it can't login! How can it find and index the millions of combinations of search results without typing in "XXXXL underpants" and all the other search combinations? Do I really need to robots.txt my search query parameter? How/why/when would googlebot generate that query parameter?
-
Site Search is B.A.D - I read this everywhere I go, but is it really? I've read - "It eats up all your search quota", "search results have no content and are classed as spam", "results pages have no value"
I want to find a positive SEO output to having a search function on my website, not just try and stifle Mr Googlebot. What I am trying to learn here is what the options are, and what are their outcomes? So far I have -
_Robots.txt - _Remove the search pages from Google
_No Index - _Allow the crawl but don't index the search pages.
_No Follow - _I'm not sure this is even a valid idea, but I picked it up somewhere out there.
_Just leave it alone - _Some of your search results might get ranked and bring traffic in.
It appears that each and every option has it's positive and negative connotations. It'd be great to hear from this here community on their experiences in this practice.
-
-
Hopefully that helps you some I know we ran into a similar situation for a client. Good luck!
-
Great idea! This has triggered a few other thoughts too... cheers Jordan.
-
I would recommend using screaming frog to crawl only product level pages and export them to a csv or excel doc then copy and past your xml sitemap into an excel sheet. Then from there I would clean up the xml sitemap and sort it by product level pages and just compare the two side by side and see what is missing.
The other option would be to go into google webmaster tools or search console and look at Google Index -> index status and then click the advanced tab and just see what is indexed and what all is being blocked by the robots.txt.
-
@jordan & @matt,
I had done this, this was my initial go-to idea and implementation, and I completely agree this is a solution.
I guess I was hoping to answer the question "can Google even use site search?". as this would answer whether the parameter even needs excluding from robots.txt (I suspect they somehow do, as there wouldn't be this much noise about it otherwise).
That leaves the current situation - Does restricting google from searching my internal search results hinder it's ability to find and index my product pages? I'd argue it does, as since implementing this 6 months ago, the site index status has gone from 5.5m to 120k.
However, this could even be a good thing, as it lowers the Googlebot activity requirement, and should focus on the stronger pages... but the holy grail I am trying to achieve here is to get all my products indexed so I can get a few hits a month from each, i'm not trying to get the search results indexed.
-
Agree with Jordan - block the parameter for search in robots.txt and forget it. It won't bring search traffic in, it shouldn't get crawled but if it does, it's always a negative.
-
I cant speak for everyone but generally we like to robots.txt the search pages. I would imagine since you are working on a large retail site you would want to ensure your other pages get indexed properly so I would imagine blocking the search pages with a robots.txt would suffice. I would also look for some common reoccuring searches through the site search to possibly build content around as well.
I hope that helps some.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Dates on Google Search Results
Hello, I manage htts://globalrose.com When I search on Google for "Yellow Roses", "Yellow Roses Globalrose", or any search that might bring up one of our pages, sometimes our search results appear with dates right before the description. Does anyone know what this mean? Why they appear on some and not other pages? Here is a search result for example: Example Google Search Can someone please help clarify this for us?
Intermediate & Advanced SEO | | globalrose.com0 -
Combining multiple HTTPS sites
Hi there! I am currently combining several sites (corporate brochure site and ecommerce site) for a client into one central website. All of the content and structure on the new site is set up and relevant pages have 301 redirects ready. My main concern is that the old .co.uk website has an SSL certificate and will be pointing to the new pages on the new .com website (with new SSL in place). Will this cause connection privacy issues? And if so, what's the best way to resolve them? Many thanks!
Intermediate & Advanced SEO | | Daniel_GlueMedia0 -
When doing a site search my homepage comes up second. Does that matter?
When I do a site: search the homepage comes up second. Does this matter?
Intermediate & Advanced SEO | | EcommerceSite0 -
Site Architecture: How flat is too flat?
There's a lot of debate out there as far as too many internal links or too many levels of a website. I've seen the videos from Rand and I've read a lot of the posts here on Moz, but I just want to know where everyone stands on this. Anyone have experience with architecture while working on a large E-commerce site? We're talking Millions of pages and over 1,000 links off the homepage alone. Anyways, I don't want to get too specific. I mostly just want to hear about experiences of the community. Thanks!
Intermediate & Advanced SEO | | GalcoIndustrial0 -
Seo flash site
Hey. Would hear whether it is possible to SEO a website which is flash site cms?
Intermediate & Advanced SEO | | Agger0 -
On-site links
Hi everybody, There's a lot of information about getting sitewide backlinks, but so few about on-site optimization. Is there a maximum of links to put on a page ? Is there a maximum of link that a page should receive ? etc ... ? So, what is the optimal strategy ? And I'm only concerned about on-page and on-site link, not backlinks commming from other sites. Thanks
Intermediate & Advanced SEO | | DavidPilon0 -
Can Someone Provide an Example of a Site that Indexes Search Results Successfully?
So, I know indexing search results is a big no-no, but I recently started working with a site that sees 50% of its traffic from search result pages. The user engagement on these pages is very high, and these pages rank well too. Unfortunately, they've been hit by Panda. They already moved the section of the site with search results to a subdomain, and saw temporary success. There must be a way to preserve their traffic from these search result pages and get out from under Panda.
Intermediate & Advanced SEO | | nicole.healthline0 -
My site has multiple H1's, one in the logo image and one as a header. Is there any official stance from the search engines on this?
In doing some research on this issue, I came across this blog post which seems to suggest it certainly will be a trigger to search engines. http://www.seounique.com/blog/multiple-h1-tags-triggers-google-penalty/ Could be a false positive on his specific case, but I was wondering what the community thought. Thanks in advance!
Intermediate & Advanced SEO | | jim_shook0