Layered navigation and hiding nav from user agent
-
I am trying to deal with the duplicate content issues presented by Magento's layered navigation feature (aka faceted navigation). I installed Amasty's Improved Navigation extension (https://amasty.com/improved-layered-navigation.html) and it offers the option to hide the layered navigation from specific user agents (ie googlebot, bingbot, etc).
This seems like cloaking to me and I hesitate to try it, unless hiding faceted navigation from specific user agents is known to be acceptable to Google (white hat practice). Does anyone know if this the case?
-
Great, thanks Carson! You're insights have been very helpful. I think we'll try to make the on-page ajax solution work.
-
If you're really worried about indexation I think that's a fine solution. It's definitely easier to manage, and it'll also be easier to track pageviews in most analytics platforms. The only downside is that if someone emails or links to a category page with filters applied the recipient won't see it. But generally people share products and not category pages, so it's not a big deal. I'd probably go that route.
Also make sure that your category pages still update the URL when you go to page 2, or that page 2 is somehow also being indexed. You don't want products that don't get indexed because categories can't be crawled.
-
Thanks for the link! I can see how Google offers me a way to tell it how to use my site variables. It seems like between managing parameters in webmaster tools, using canonical links and adding meta noindex tags on variable pages, there should be enough to keep things in order with the search engines. And I can just assume Google knows not to waste too much crawl budget on the variable pages.
I was considering one other option that would remove concerns about variables altogether. Using a different extension, I can set up Magento's layered navigation to work on the page without updating the URL. This eliminates the need for canonicals, parameters, and everything else that is more in Google's control than mine. What do you think of that as a solution?
-
Yes, the bots will crawl the pages, but they will not INDEX them.
There is a concern there, but mostly if the bots get caught in some kind of crawl trap - where they're trying out a near-infinite set of variables and getting stuck in a loop. Otherwise the spiders should understand the variables. You can actually check it in Webmaster tools to make sure Google understands. Instructions for that here:
https://support.google.com/webmasters/answer/6080550?hl=en
Ultimately Google will definitely not penalize you for having lots of duplicate content on URLs through variables, but it might be an issue with Googlebot not finding all your pages. You can make sure that doesn't happen by checking the indexation of your sitemap.
You could also try to block any URLs with the URL parameter in robots.txt. Make sure you get some help on the RegEx if you plan to do this. My advice is that blocking the variables in robots.txt is not worth it, as Google should have no problems with the variables - especially if the canonical tags are working.
Googlebot at least is smart enough these days to know when to stop crawling variable pages, so I think there are more important on-site things to worry about. Make sure your categories are linked to and optimized, for example.
-
This gets into an issue of bots and crawling where I am less clear. Even with canonicals, don't search engine bots crawl all of the pages produced with faceted navigation? That will easily reach 10,000+ pages on my site, which currently has a total number of pages in the low hundreds. I was under the impression I don't want to set up the faceted navigation in a way where the bots crawl through every combination of pages created by my products' attribute filters and bog the bots down in a quagmire of low-value pages. But I'm not sure if that's the case or how concerned I need to be about the bots spending their time on those pages.
-
If I'm not mistaken Magento has canonical tags on category pages by default, so you might be trying to solve an issue that doesn't exist. Take a look at the source code on faceted navigation to confirm. Or you can send me the site and I'll look over it.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
The main navigation is using JS, will this have a negative impact on SEO?
Hi mozzers, We just redesigned our homepage and discovered that our main nav is using JS and when disabling JS, no main nav links was showing up. Is this still considered bad practice for SEO? https://cl.ly/14ccf2509478 thanks
Intermediate & Advanced SEO | | Ty19861 -
HREF LANG: Different navigation/structure per country: is that a problem?
Hi all, One question about the href lang tag. Our webshop sells to 4 different countries (the Netherlands, Germany, Belgium & Spain). The navigation is a little bit different for these countries, depending on how popular certain product categories are in certain countries. So, for example: Netherlands --> Category A and B are in the top navigation
Intermediate & Advanced SEO | | AMAGARD
Germany --> Category B is a subcategory of product A. We want to implement the Hreflang tag, would it be a problem that the navigation/site structure (and therefore the URL structure for certain categories) are a bit different? So: The url for category B in the Netherlands is: https://www.website.com/nl/category-B/
The url for category B in Germany is: https://www.website.com/de/category-A/category-B/ Thanks in advance! Best!0 -
Tricky: Should I remove this extra navigation?
Hello there, I am working on a website in which the owner has written navigation links manually at the end of all pages/posts. Example. Page/Post Entry content. Go to home click here,
Intermediate & Advanced SEO | | Ganacontrol1233
Menu item 2
Menu item 3
Menu item 4 etc We already have navigation links for that on the main menu and additionally on a sidebar. But we are two years in, and they have always been there. Is there a chance removing all these links will better internal pagerank distribution? Website has millions of views/month so I want to be sure - if I should just leave them as is or remove them all.
Or what extra questions should I be asking myself about this.0 -
Navigation is not showing up when disabling JS?
HI Mozzers, I am checking at a site Navigation and when disabling JS, I can see that navigation is not present. When checking at individual nav items, they are true HTML I believe since they carry attribute but somehow not crawlable. I am confused since JS is disabled and Nav is not showing up, isn't it suppose to mean that Nav is 100% JS? Can someone tell if the Nav is good or bad based on the fact it has true HTML? I am not sure because couple years ago I knew that Google was not crawling well JS but seems that things have change based on http://searchengineland.com/tested-googlebot-crawls-javascript-heres-learned-220157. Thanks for letting me know!
Intermediate & Advanced SEO | | Ideas-Money-Art0 -
How does Tripadviser ensure all their user reviews get crawled?
Tripadvisor has a LOT of user generated content. Searching for a random hotel always seems to return a paginated list of 90+ pages. However once the first page is clicked and "#REVIEWS" is appended to the URL, the URL never changes with any subsequent clicks of the paginated links. How do they ensure that all this review content gets crawled? Thanks, linklater
Intermediate & Advanced SEO | | linklater0 -
Pagination and View All Pages Question. We currently don't have a canonical tag pointing to View all as I don't believe it's a good user experience so how best we deal with this.
Hello All, I have an eCommerce site and have implemented the use rel="prev" and rel="next" for Page Pagination. However, we also have a View All which shows all the products but we currently don't have a canonical tag pointing to this as I don't believe showing the user a page with shed loads of products on it is actually a good user experience so we havent done anything with this page. I have a sample url from one of our categories which may help - http://goo.gl/9LPDOZ This is obviously causing me duplication issues as well . Also , the main category pages has historically been the pages which ranks better as opposed to Page 2, Page 3 etc etc. I am wondering what I should do about the View All Page and has anyone else had this same issue and how did they deal with it. Do we just get rid of the View All even though Google says it prefers you to have it ? I also want to concentrate my link juice on the main category pages as opposed being diluted between all my paginated pages ? - Does anyone have any tips on how to best do this and have you seen any ranking improvement from this ? Any ideas greatly appreciated. thanks Peter
Intermediate & Advanced SEO | | PeteC120 -
Hiding answers
I would like to create a page with a few questions and when people click the answer comes up. Is google going to index and rank for the hidden text that is comming up when clicking?
Intermediate & Advanced SEO | | Joseph-Green-SEO0 -
Does hiding responsive design elements on smaller media types impact Google's mobile crawler?
I have a responsive site and we hide elements on smaller media types. For example, we have an extensive sitemap in the footer on desktop, but when you shrink the viewport to mobile we don't show the footer. Does this practice make Google's mobile bot crawler much less efficient and therefore impact our mobile search rankings?
Intermediate & Advanced SEO | | jcgoodrich1