On-site Search - Revisited (again, *zZz*)
-
Howdy Moz fans!
Okay so there's a mountain of information out there on the webernet about internal search results... but i'm finding some contradiction and a lot of pre-2014 stuff. Id like to hear some 2016 opinion and specifically around a couple of thoughts of my own, as well as some i've deduced from other sources. For clarity, I work on a large retail site with over 4 million products (product pages), and my predicament is thus - I want Google to be able to find and rank my product pages. Yes, I can link to a number of the best ones by creating well planned links via categorisation, silos, efficient menus etc (done), but can I utilise site search for this purpose?
-
It was my understanding that Google bots don't/can't/won't use a search function... how could it? It's like expeciting it to find your members only area, it can't login! How can it find and index the millions of combinations of search results without typing in "XXXXL underpants" and all the other search combinations? Do I really need to robots.txt my search query parameter? How/why/when would googlebot generate that query parameter?
-
Site Search is B.A.D - I read this everywhere I go, but is it really? I've read - "It eats up all your search quota", "search results have no content and are classed as spam", "results pages have no value"
I want to find a positive SEO output to having a search function on my website, not just try and stifle Mr Googlebot. What I am trying to learn here is what the options are, and what are their outcomes? So far I have -
_Robots.txt - _Remove the search pages from Google
_No Index - _Allow the crawl but don't index the search pages.
_No Follow - _I'm not sure this is even a valid idea, but I picked it up somewhere out there.
_Just leave it alone - _Some of your search results might get ranked and bring traffic in.
It appears that each and every option has it's positive and negative connotations. It'd be great to hear from this here community on their experiences in this practice.
-
-
Hopefully that helps you some I know we ran into a similar situation for a client. Good luck!
-
Great idea! This has triggered a few other thoughts too... cheers Jordan.
-
I would recommend using screaming frog to crawl only product level pages and export them to a csv or excel doc then copy and past your xml sitemap into an excel sheet. Then from there I would clean up the xml sitemap and sort it by product level pages and just compare the two side by side and see what is missing.
The other option would be to go into google webmaster tools or search console and look at Google Index -> index status and then click the advanced tab and just see what is indexed and what all is being blocked by the robots.txt.
-
@jordan & @matt,
I had done this, this was my initial go-to idea and implementation, and I completely agree this is a solution.
I guess I was hoping to answer the question "can Google even use site search?". as this would answer whether the parameter even needs excluding from robots.txt (I suspect they somehow do, as there wouldn't be this much noise about it otherwise).
That leaves the current situation - Does restricting google from searching my internal search results hinder it's ability to find and index my product pages? I'd argue it does, as since implementing this 6 months ago, the site index status has gone from 5.5m to 120k.
However, this could even be a good thing, as it lowers the Googlebot activity requirement, and should focus on the stronger pages... but the holy grail I am trying to achieve here is to get all my products indexed so I can get a few hits a month from each, i'm not trying to get the search results indexed.
-
Agree with Jordan - block the parameter for search in robots.txt and forget it. It won't bring search traffic in, it shouldn't get crawled but if it does, it's always a negative.
-
I cant speak for everyone but generally we like to robots.txt the search pages. I would imagine since you are working on a large retail site you would want to ensure your other pages get indexed properly so I would imagine blocking the search pages with a robots.txt would suffice. I would also look for some common reoccuring searches through the site search to possibly build content around as well.
I hope that helps some.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why does some sites rank with no seo
Why is it that some site rank with zero efforts? I have been working on some seo for a while on my main site and i have been getting more info competition analysis with sem and moz. Looking at the states from this website which tends to popup often in the searches on page 1-2 before my site. This site is not keyword optimized, meaning they arent even trying to rank.
Intermediate & Advanced SEO | | CooperStrzelecki
There is no content, articles etc.,
6 backlinks (nothing powerful just 2 directory links and 2 from developer)
Site really isnt even designed to get traffic as its a trade only ecommerce website
I doubt they are hiding anything as far as backlinks etc. as it will get them too many visitors they dont want
The city i am searching isnt even on the page (it is a city within a city so maybe google still relates it)
PA 24 DA 15 Now my site:
Optimized reasearched keywords
175 backlinks
All my main pages have content with images, alt tags, internal linking
full of content, blogs, videos, products (probably 4000, could a site being too big be an issue?)
Site gets regular updates
I probably have 200 citations
All the social media which gets done often
PA 32 DA 20 They do get a good bit of traffic but that is probably the only thing i would see but it would be direct traffic mostly i believe as it would be people going to order regularly since it is a print reseller. They may have some age on me 15 vs 8 years. Could it be some kind of penalty i am not sure about lingering? According to what i know to check everyything looks ok, no shady links accoding to sem. I am working more and more on all the pages but this competittion site really doesnt have crap going on probably 8 pages and 1 page does all the ordering. What the hell does google want from me exactly!0 -
How to avoid adult traffic to site?
A client of ours is increasingly getting a lot of adult traffic to their site, where they show up only for adult searches and not at all for relevant searches. How can we stop Google associating their site with adult content? Here's a blog example, giving advice to parents on girls and body image issues: https://www.commonsensemedia.org/blog/girls-and-body-image keywords driving traffic to this page are all around images for 'young nude girls' etc.
Intermediate & Advanced SEO | | MediaCause0 -
Open Site Explorer - Spam analysis: need help with inbound links... from my site!
hallo, reading my spam analysis report from open explorer, I found somenthing I don't understand (please see attached image): The long list of links inside the red rectangle are inbound links with a spam score of 5 coming from my same site. How is that possible? Should I remove those links? Also , I see that many of those links are links present in the top navigation bar (about page, home page, service description etc.) or in the sidebar section of the website (categories, recent posts, recent comments). Should I treat them differently? Thank you for your time.
Intermediate & Advanced SEO | | micvitale0 -
Do I have to optimize every page on my site?
Hi guys I run my own photography webstie (www.hemeravisuals.co.uk Going through the process optimizing my page for seo. I have one question I have a few gallery pages with no text etc? Do I still have to optimize these ? Would it rank my site lower if they weren't optimized? And how can i do this sucessfully with little text on these pages ( I have indepth text on these subjects on my services & pricing pages? Kind Regards Cam
Intermediate & Advanced SEO | | hemeravisuals0 -
Google Sitelinks Search Box
For some reason, a search for our company name (“hometalk”) does not produce the search box in the results (even though we do have sitelinks). We are adding schema markup as outlined here, but we're not sure about: Will adding the code make the search bar appear (or at least increase the chances), or is it only going to change the functionality of the search box (to on-site search) for results that are already showing a search bar?
Intermediate & Advanced SEO | | YairSpolter0 -
Indexing an e-commerce site
Hi all, My client babyblingstreet.com. She sells baby and toddler clothing. Now a lot of the links on her site contain the same products. For instance: if you go to "What's new" you can find those same products in let's say her "Sale Items" link category. The real problem with this is let's say my client sells a green dress and someone accesses it through the "baby and toddler dresses" category. And let's say this URL has 10 links pointing to it. Now, let's say someone else accesses this same green dress through the "What's new" category. And let's say this particular URL has 10 links pointing to it. Instead of having 20 links pointing to one URL about the green dress, I now have 10 links pointing to one URL and 10 pointing to another URL even though both URLs feature the exact same green dress. In this particular example I would want to make the URL of the green dress in the "baby and toddler clothing" section be the canonical URL. So that means I would have to use this canonical tag on the green dress URL that's in the "what's new" category and let's say also the "sale items" category. This could get very tedious if my client has 200+ products. So I am wondering if I have to place a canonical tag on every URL that displays the green dress? More importantly, I would like to know other people's strategies for indexing e-commerce sites that have the same product featured in multiple categories throughout the site. I hope this makes sense. Thanks for your time.
Intermediate & Advanced SEO | | jenga110 -
Penalties for site going down often?
I have a client with a site that ranks for some very competitive terms who consistently has server issues and the site goes down for a day at a time. Each time this happens his site seems to drop in site wide rankings and then stay there for months without ever fully recovering. Only part of the rankings are usually recovered. Has anyone else seen this trend? Is it something Google keeps on record without fully removing any penalty addressed?
Intermediate & Advanced SEO | | iAnalyst.com0 -
Migrating a site
Hello, I have what a I think it's a noob question.. I have a medium size website and need to put it into maintenance for the next 2 months, and afterwards activate a completly new site. My client asked me to do this, cause the same people whoe run the constant flow of information on the site, are the ones who are going to develop the new site, so he wants to just close it out So... what are the steps for doing this with minimum impact on any SEO advances made this past months?.. How do I tell the search engines, Hey, just under maintenance for a while....then... i'm back in the game but this is my new structure. and the old one should go here
Intermediate & Advanced SEO | | daniel.alvarez0