Drop Down Menus and Crawlability
-
Hello,
We are working on a complete site redesign. One of the mock-ups that are being reviewed is of a page that encompasses and entire category of products, but the only way the user can see the products is to fill out several drop down menus, and then a subset of products that match that criteria will appear.
Once that list appears, the user will then be able to click on each of the products and will then be taken to the product page.
I'm concerned that this layout will pose a crawlability issue since click activity and drop down menus have always been a problem for bots in the past, has anything changed? Will the bot be able to follow the links to these product pages if it can't see them since it can't fill out the form?
Also, depending on the functionality of this 'form', I'm assuming the product listing will be populated dynamically and pulled from another source, which means that the product links will not live in the html of the page, and hence cannot be crawled. Does anyone know how this is normally handled? Do the actual results usually live elsewhere or does it live in the html of that page?
Any thoughts or clarity around this would be appreciated.
-
"But if they are already in the html, would that be considered cloaking?"
There are times when presenting something in HTML that is otherwise not visible, but having other features on the page that allow people to read / click / access that content in other ways, is fine. Linking is a tricky because links are so inherently valuable in terms of SEO. You don't can't really be too careful.
I'd be wary of presenting links (a subset or full set) in HTML if there is a form process to actually arrive at the links' targets. Essentially you'll be linking to products X, Y and Z on a page, for search engines but requiring a specific input from a user to see X, Y or Z - an input that only very few overall visitors are actually likely to make. I would say this qualifies as showing different content for SEO's sake and not providing a UX alternative that is pretty much the same thing. Others may disagree with me on that - I'm being wary here
I would very much like to see the HTML if you are still active in this thread when it is produced, but you may be left with a situation where the pages need to be linked to elsewhere throughout the site to ensure they are crawled.
-
Thanks, Jane. I don't have the raw html because only a static design has been produced at this point.
I'm not sure I want the form filled out by the bots, I just want to make sure that the links that are the end result of filling out the form are crawlable because that will be the only path to the product pages. I've been speaking with IT to figure out if the links will already be in the html even if they are not displayed on the page, or if the links are dynamically generated from another location, which means they will not be crawlable. They are not sure yet. But if they are already in the html, would that be considered cloaking? Since the user cannot see them until they fill out the form? And even then they will only see a small subset of the links.
My other concern for this page is that we are taking our largest parent category and putting ALL of the products on one page - you just can't get to them until you fill out the form. My worry is that this page will be way to broad, this parent category is normally made up of several subcategories. I don't think we will rank well for some long tail terms because there is going to be so much broad content on the page pertaining to so many different types of products.
Any thoughts or suggestions are appreciated.
-
Hi Kelli,
From what you have described here, I don't think this will easily be crawled. Obviously the necessary code for the click activity is included in the HTML, and whilst Google has been known to complete forms and take actions on sites before, it's far from guaranteed that it will do this.
Usually when Google completes actions like this, it's not desirable - you used to see websites with millions of junk URLs indexed because Google had either "filled out forms" itself, or spammers had dynamically generated millions of versions of a form to fill Google's index, in order to harm a competitor. It's not common to want Google to complete activity like this, rather than just let it crawl the information deliberately given to it in HTML.
I would be really curious to see what the menus looked like in HTML though. That would give us a better idea of whether it's likely Google will crawl any of the content.
If the menus are not crawlable, there are range of other good options (that can also be user-friendly and attractive) for menu development. The Distilled guide is a good resource.
If we are I am able to look at the raw HTML for the planned menus, please let me know. If you'd rather not post it in here, feel free to PM me (I am not a Moz staff member - I used to be - but I act as an associate contractor for the company) or email jane.copland@gmail.com.
Cheers,
Jane
-
Thanks Evan. Please keep in mind, this is not the navigation, it is essentially a form on a page that dynamically generates a list of product page links. My question is that I want to know if those products cannot be viewed until the form is filled out, how can the bots see them.
This form will require click activity to fill out, not just hovering over it. And I don't just want the dropdowns to be crawled, the dropdown items themselves are not links, they are just decisions that the user has to make in order to see matching products.
Even if the bot could fill out the form, it is only going to display a small subset of product links. If this is the only page that will have links to all of our products in a particular category, I want to make sure that all of those product pages will get crawled. So I was wondering if all of the product links will still be see by the bots even though the user will not be able to see them.
-
hey Kelli,
i'm not entirely sure what the mock-up design is like, but I have used dropdown me us in the past, and as long as they are in html, bots should be able to crawl. I have found this article helpful on the past.: https://www.distilled.net/blog/seo/site-navigation-for-seo/
Hopefully this is helpful.
-
Thanks, but I cannot fetch as googlebot because the page is not live yet, we are wireframing the design first.
-
A simple way to see how Google sees your page is to use the "Fetch as Googlebot" function in Google Webmasters. This way you can see if there is anything not being crawled. The more traditional way to do this set up would be to have a search bar above the fold, and then have category pages people can click through to browse if they want. Messy drop-downs are never fun.
Let me know if that helps.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Website Drops Some Traffic after Redesign. What's Happening?
What it is NOT: No Link was broken. I have used Moz, Screaming Frog, Excel, etc - there are not broken links. We have not added spammy links. We kept the same amount of links and content on the homepage - with an exception of 1 or 2. All the pages remained canonical. Our blog uses rel=prev rel=next, and each page is canonicalized to itself. We do not index duplicated content. Our tags are content="noindex,follow" We are using the Genesis Framework (we were not before.) Load time is quicker - we now have a dedicated server. Webmaster tools has not reported any crawl report problems. What we did that should have improved our rankings and traffic: Implemented schema.org Responsive design Our bounce rate is down - Average visit length is up. Any ideas?
Web Design | | Thriveworks-Counseling0 -
Website design chnage and massive traffic drop?
I changed my Drupal theme to "Bootstrap 3" and now my traffic is down 50% gradually in past 5 days, can this be theme related? Answers to checks below : 1). There were no redirects involved, I just flipped a switch and changed theme for my Drupal blog. 2). No issues reported by Google WMT except the fact that impressions fell, see stat images for comparison at - http://imgur.com/a/5PssH#0. 3). site:mysiteurl.com shows healthy "About 201,000 results". 4). Checked slow loading times and browser issue, nothing there. 5). It's not a seasonal drop. Pls. suggest what else should I focus upon to find the reason. @Prateek_Chandra where can I share my analytic report with you privately. I can also enable guest access to my account for you to have a look.
Web Design | | techdna11 -
Domain Authority Drop After Website Relaunch
Prior to my website redesign and relaunch on July 10th, our domain authority was 33. 301 redirects were implemented properly. Out or 600 pages, about 200 URLs were modified. Domain authority has dropped to 28. Rankings are terrible. Conversions are awful. What does the domain authority drop mean? I have noticed in the past that a drop in domain authority tends to coincide with more a drop in ranking and a drop in the quality of visitors. The site is www.metro-manhattan.com Thanks,
Web Design | | Kingalan1
Alan0 -
Could our drop in organic rankings have been caused by improper mobile site set-up?
Site: 12 year old financial service 'information' site with lead gen business model. Historically has held top 10 positions for top keywords and phrases. Background: The organic traffic from Google has fallen to 50% of what it was over the past 4 months compared to the same months last year. While several potential factors could be responsible/contributing (not limited to my pro-active removal of a dozen old emat links that may be perceived as unnatural despite no warning), this drop coincides with the same period the 'mobile site' was launched. Because I admittedly know the least about this potential cause, I am turning to the forum for assistance. Because the site is ~200 pages and contains many 'custom' pages with financial tables, forms, data pulled from 3rd parties, custom/different layouts we opted for creating a mobile site of only the top 12 most popular pages/topics just to have a mobile presence (instead of re-coding the entire site to make it responsive utilizing a mobile css). -These mobile pages were set up in an "m." subdomain. -We used bi-directional tagging placing a rel=canonical tag on the mobile page, and a rel=alternate tag on the desktop page. This created a loop between the pages, as advised by Google. -Some mobile pages used content from a sub page, not the primary desktop page for a particular topic. This may have broken the bi-directional 'loop', meaning the rel=canonical on the mobile page would point to a subpage, where the rel=alternate would point to the primary desktop page, even though the content did not come from that page, necessarily. The primary desktop page is the one that ranks for related keywords. In these cases, the "loop" would be broken. Is this a cause for concern? Could the authority held by the desktop page not be transferred to the mobile version, or the mobile page 'pull away' or disperse the strength of the desktop page if that 'loop' was not connected? Could not setting up the bi-directional tags correctly cause a drop in the organic rankings? -Our developer verified the site is set up according to Google's guidelines for identifying device screen size and serving appropriate version of page. -Are there any tools or utilities that I can use to identify issues, and/or verify everything is configured correctly? -Are we missing anything important in the set-up/configuration? -Could the use of a brand new subdomain 'm.' in and of itself be causing issues? -Have I identified any negative seo practices or pitfalls? Am I missing or overlooking something? While i would have preferred maintaining a single, responsive, site with mobile css, it was not realistic given the various layouts, and owner's desire to only offer the top pages in mobile format. The mobile site may have nothing to do with the organic drop, but I'd like to rule it out if so, and I have so many questions. If anyone could address my concerns, it would be greatly appreciated. Thanks! Greg
Web Design | | seagreen0 -
New more "helpful" internal linking causing SERP & traffic drop?
Still dealing with the weird traffic drop on my website. I have removed a bunch of old links from a defunct blog, 301 thin pages, added text to remaining pages. I'm still stumped. So awhile ago I freshened up my website and thought I was "helping people" by making sure they could CONTACT the studio more easily... I added more links to the "contact page" I thought this would help conversions...This changed the number of links to my entire site....Would this be the problem with my ranking/traffic drop? http://bayareaboudoir.com/babinternal1.pdf
Web Design | | Squee1 -
Penguin 2.0 drop due to poor anchor text?
Hi, my website experienced a 30% drop in organic traffic following the Penguin 2.0 update, and after years of designing my website with SEO in mind, generating unique content for users, and only focusing on relevant websites in my link building strategy, I'm a bit disheartened by the drop in traffic. Having rolled out a new design of my website at the start of April, I suspect that I've accidentally messed up the structure of the website, making my site difficult to crawl, or making Google think that my site is spammy. Looking at Google Webmaster Tools, the number 1 anchor text in the site is "remove all filters" - which is clearly not what I want! The "remove all filters" link on my website appears when my hotels page loads with filters or sorting or availability dates in place - I included that link to make it easy for users to view the complete hotel listing again. An example of this link is towards the top right hand side of this page: http://www.concerthotels.com/venue-hotels/agganis-arena-hotels/300382?star=2 With over 6000 venues on my website, this link has the potential to appear thousands of times, and while the anchor text is always "remove all filters", the destination URL will be different depending on the venue the user is looking at. I'm guessing that to Google, this looks VERY spammy indeed!? I tried to make the filtering/sorting/availability less visible to Google's crawl when I designed the site, through the use of forms, jquery and javascript etc., but it does look like the crawl is managing to access these pages and find the "remove all filters" link. What is the best approach to take when a standard "clear all..." type link is required on a listing page, without making the link appear spammy to Google - it's a link which is only in place to benefit the user - not to cause trouble! My final question to you guys is - do you think this one sloppy piece of work could be enough to cause my site to drop significantly following the Penguin 2.0 update, or is it likely to be a bigger problem than this? And if it is probably due to this piece of work, is it likely that solving the problem could result in a prompt rise back up the rankings, or is there going to be a black mark against my website going forward and slow down recovery? Any advice/suggestions will be greatly appreciated, Thanks Mike
Web Design | | mjk260 -
Can "poor" subdomains drop PR of the root domain?
The page rank of my company's website has dropped from a 6 to a 4 over the past year or so. In that time, we implemented subdomains for development sites to show clients progress on their websites. I noticed that our "dev" sites are being indexed while in development and my question is, will Google drop pagerank of our root domain purely off of these "dev" subdomains? Example - our site is www.oursite.com Dev site - development1.oursite.com I just began investigating the drop and this came to my mind yesterday but am not too sure what type of impact these non-credible subdomains will have on our root domain. Any thoughts?
Web Design | | ckilgore0 -
Google News we were dropped and need help finding ot why
Hi i have a site called in2town lifestyle magazine http://www.in2town.co.uk/ and up until two months ago we were with google news and for a long time. But then all of a sudden we were dropped which left us with no confidence about our site and led us to make changes to the site, some good and some bad to try and find out what was wrong with our site and why we were dropped. We have now been concentrating on sorting the site out which has led in a drop in traffic due to not updating it as we should because we are more concerned in trying to make it a quality lifestyle magazine and get back in google as well as making it a good experience for our readers.. I would like your help and finding out what you feel is wrong with our site so we can then work on it and change it and try and find out what went wrong with google news. we have spent years on the site but now we have gone in the wrong direction because we were more worried about google news. If you can advise us on how we should change the site and sort the site out and make it into the professional site it was once more then that would be great.
Web Design | | ClaireH-1848860