On-site Search - Revisited (again, *zZz*)
-
Howdy Moz fans!
Okay so there's a mountain of information out there on the webernet about internal search results... but i'm finding some contradiction and a lot of pre-2014 stuff. Id like to hear some 2016 opinion and specifically around a couple of thoughts of my own, as well as some i've deduced from other sources. For clarity, I work on a large retail site with over 4 million products (product pages), and my predicament is thus - I want Google to be able to find and rank my product pages. Yes, I can link to a number of the best ones by creating well planned links via categorisation, silos, efficient menus etc (done), but can I utilise site search for this purpose?
-
It was my understanding that Google bots don't/can't/won't use a search function... how could it? It's like expeciting it to find your members only area, it can't login! How can it find and index the millions of combinations of search results without typing in "XXXXL underpants" and all the other search combinations? Do I really need to robots.txt my search query parameter? How/why/when would googlebot generate that query parameter?
-
Site Search is B.A.D - I read this everywhere I go, but is it really? I've read - "It eats up all your search quota", "search results have no content and are classed as spam", "results pages have no value"
I want to find a positive SEO output to having a search function on my website, not just try and stifle Mr Googlebot. What I am trying to learn here is what the options are, and what are their outcomes? So far I have -
_Robots.txt - _Remove the search pages from Google
_No Index - _Allow the crawl but don't index the search pages.
_No Follow - _I'm not sure this is even a valid idea, but I picked it up somewhere out there.
_Just leave it alone - _Some of your search results might get ranked and bring traffic in.
It appears that each and every option has it's positive and negative connotations. It'd be great to hear from this here community on their experiences in this practice.
-
-
Hopefully that helps you some I know we ran into a similar situation for a client. Good luck!
-
Great idea! This has triggered a few other thoughts too... cheers Jordan.
-
I would recommend using screaming frog to crawl only product level pages and export them to a csv or excel doc then copy and past your xml sitemap into an excel sheet. Then from there I would clean up the xml sitemap and sort it by product level pages and just compare the two side by side and see what is missing.
The other option would be to go into google webmaster tools or search console and look at Google Index -> index status and then click the advanced tab and just see what is indexed and what all is being blocked by the robots.txt.
-
@jordan & @matt,
I had done this, this was my initial go-to idea and implementation, and I completely agree this is a solution.
I guess I was hoping to answer the question "can Google even use site search?". as this would answer whether the parameter even needs excluding from robots.txt (I suspect they somehow do, as there wouldn't be this much noise about it otherwise).
That leaves the current situation - Does restricting google from searching my internal search results hinder it's ability to find and index my product pages? I'd argue it does, as since implementing this 6 months ago, the site index status has gone from 5.5m to 120k.
However, this could even be a good thing, as it lowers the Googlebot activity requirement, and should focus on the stronger pages... but the holy grail I am trying to achieve here is to get all my products indexed so I can get a few hits a month from each, i'm not trying to get the search results indexed.
-
Agree with Jordan - block the parameter for search in robots.txt and forget it. It won't bring search traffic in, it shouldn't get crawled but if it does, it's always a negative.
-
I cant speak for everyone but generally we like to robots.txt the search pages. I would imagine since you are working on a large retail site you would want to ensure your other pages get indexed properly so I would imagine blocking the search pages with a robots.txt would suffice. I would also look for some common reoccuring searches through the site search to possibly build content around as well.
I hope that helps some.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
My site is being deindexed for unknown reason
A few days ago I noticed that my site gusty.se was not showing up in google, only the subpages. There is no message in the google search console. I requested the site to be reindexed and about a day later the site was showing up in google again. Now another day has past and the site is now again not indexed in google. Question is why the site is being deindexed??? I have worked a bit with getting backlinks to the site and I did recently gain 3 backlinks within a few days (about a week has past since I gained these links). Still I can't believe Google would count this as unnatural link building, especially since I guess it will take some time for Google to detect new incoming links. Another thing I've notice though is that my site about two weeks ago got a high number of incoming links from different spam sites with .gq TLD's (see the attached screenshot). The majority of these sites have however not linked to my main page but to a sub page which still is indexed by Google. Can all these spamlinks be the reason to why Google has deindexed the main page of my site? I've read that Google in general ignore links from spam sites, still I have taken action against these spam sites by submitting a disavow text file containing all these spam domains. I submitted this file about 2 days ago. I have now again requested the site to be reindexed so perhaps will it soon be listed again. Still, I can't keep having my site deindexed and having me reindexing it every second day. I would really appreciate if someone could give me some insight in this problem. moz.jpg
Intermediate & Advanced SEO | | Grodan21 -
Which search engines should we submit our sitemap to?
Other than Google and Bing, which search engines should we submit our sitemap to?
Intermediate & Advanced SEO | | NicheSocial0 -
An improved search box within the search results - Results?
Hello~ Does anyone have any positive traffic results to share since implementing this? Thanks! MS
Intermediate & Advanced SEO | | MargaritaS0 -
ECommerce search results to noindex?
Hi, To avoid duplicated content and the possibility of thousands additional pages to an ecommerce website would it be a reasonable solution to have the page as a no-index, would this benefit the site? Thanks **Lantec **
Intermediate & Advanced SEO | | Lantec0 -
Why Did My Site Go Limp On Me?
One of my clients was once in the #1 position for "Philadelphia interior designer" and other related terms, but her site has dropped significantly. Still it is on the first page, but far from its former glory. http://www.interiorsbydonnahoffman.com is the site. What really confuses me is why in her home turf search of "Bucks County Interior Designer" a competitor, http://www.miriamansellinteriors.com, is above her in the SERPS. According to OSE her competitor has a PA of 32 vs my client's 39. My client has 35 Linking Root Domains (and some of high quality) compared to just 11 for the competition. In all aspects her competitor looks weaker and less relevant to me. Her site has been weak in the SERPs since May/June. We are redesigning her site- she has a high bounce rate compared to my other interior design clients, something like 55%. Any insights from y'all?
Intermediate & Advanced SEO | | dfhytrwy0 -
New Site Structure and 301s
We're moving towards a new site with new site structure. The old site has numerous backlinks to past events that won't be published on the new site. The new site will have about 60 future events that are currently active on the old site as well. I was wondering the best way to move forward with the 301 redirect plan. I was considering redirecting the old site structure to an "archive.ourdomain.co.uk" subdomain and redirecting the 60 or so active events to their equivalents on the new site. Would this be a sensible plan? Also for the active events, is there any difference between: _redirecting the old page to the archive page and then forwarding to the equivalent on the new page _ and redirecting the old page directly to the new page
Intermediate & Advanced SEO | | chanm790 -
301 entire site
Is there a good 301 code snippet to change just the root domain but keep the ending extensions? I just bid on a domain that I think would be much better for me moving forward, but do not want to have to try going through thousands of pages to do their 301 individually My site is almost 4 yrs old. Well established and has a large fanbase. Several of our social networks are under the name of the new branded domain, hence part of the desire to switch.
Intermediate & Advanced SEO | | Atomicx0 -
Places Listing in Search Results
Hi everyone, We have a company that hired us to set-up their Google Places listing for their 2nd location. The listing for the 1st location is very strong. Lots of reviews, Zagat rating, Knowledge graph, etc. In the search results the Google Places listing for the 1st location has merged with the website listing. You can see the link to the main site w/ a small grey google places listing directly below it. The client would like BOTH Google places listing to show up in the search results. They both show up on the map listing but not in the search results. Each location has its own listing in Google Places. We have also created different pages on the website for each location. Is there a way to get the search results to display places listings? I have noticed a few other business have done it by naming each of their multiple locations something slightly different. Then the search results seem to realize there are multiple locations and display the places listing in the search results. Anyone run into this? Any ideas? Thanks!!
Intermediate & Advanced SEO | | SeattleJoe0