Crawl Budget and Faceted Navigation
-
Hi, we have an ecommerce website with facetted navigation for the various options available.
Google has 3.4 million webpages indexed. Many of which are over 90% duplicates.
Due to the low domain authority (15/100) Google is only crawling around 4,500 webpages per day, which we would like to improve/increase.
We know, in order not to waste crawl budget we should use the robots.txt to disallow parameter URL’s (i.e. ?option=, ?search= etc..). This makes sense as it would resolve many of the duplicate content issues and force Google to only crawl the main category, product pages etc.
However, having looked at the Google Search Console these pages are getting a significant amount of organic traffic on a monthly basis.
Is it worth disallowing these parameter URL’s in robots.txt, and hoping that this solves our crawl budget issues, thus helping to index and rank the most important webpages in less time.
Or is there a better solution?
Many thanks in advance.
Lee.
-
Hello, I have also been in a similar situation. What I did was to disallow the urls with parameters using the robots.txt and place (in only the pages with parameters) the following two html tags:
This will expressly indicate to google not to index these pages. I still have some errors but I guess they will disappear in a few months.
Regards
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
When "pruning" old content, is it normal to see an drop in Domain Authority on Moz crawl report?
After reading several posts about the benefits of pruning old, irrelevant content, I went through a content audit exercise to kick off the year. The biggest category of changes so far has been to noindex + remove from sitemap a number of blog posts from 2015/2016 (which were very time-specific, i.e. software release details). I assigned many of the old posts a new canonical URL pointing to the parent category. I realize it'd be ideal to point to a more relevant/current blog post, but could this be where I've gone wrong? Another big change was to hide the old posts from the archive pages on the blog. Any advice/experience from anyone doing something similar much appreciated! Would be good to be reassured I'm on the right track and a slight drop is nothing to worry about. 🙂 If anyone is interested in having a look: https://vivaldi.com https://vivaldi.com/blog/snapshots [this is the category where changes have been made, primarily] https://vivaldi.com/blog/snapshots/keyboard-shortcut-editing/ [example of a pruned post]
Intermediate & Advanced SEO | | jonmc1 -
How to remove skip links, main navigation, sidebars as h2 tags in wordpress genesis
Our website CMS is wordpress. Due to the Genesis Framework; below 4 phrases tuned into h2 tags: Skip links, Header Right, Main navigation and Footer. How to remove these?
Intermediate & Advanced SEO | | vtmoz0 -
Is there anything I need to worry about if... We show/hide header navigation based upon visit from external traffic?
Scenario: So imagine if LinkedIn turned off their main navigation/header if you landed on your personal profile via a search engine or via an external link. But if you were on LinkedIn when you found it, the navigation remains the same.
Intermediate & Advanced SEO | | mysitesrock0 -
Does Google crawl and spider for other links in rel=canonical pages?
When you add rel=canonical to the page, will Google still crawl your page for content and discover new links in that page?
Intermediate & Advanced SEO | | ReferralCandy0 -
Are Navigation links different to static links
We are trying to reduce the number of links on our homepage. We could remove some fly out navigation links, We rank 1st on Google for some of these links. Would removing these hurt our SEO. The links are accessible 1 level down if we remove the homepage.
Intermediate & Advanced SEO | | Archers0 -
Issues with Google-Bot crawl vs. Roger-Bot
Greetings from a first time poster and SEO noob... I hope that this question makes sense... I have a small e-commerce site, I have had Roger-bot crawl the site and I have fixed all errors and warnings that Volusion will allow me to fix. Then I checked Webmaster Tools, HTML improvements section and the Google-bot sees different dupe. title tag issues that Roger-bot did not. so A few weeks back I changed the title tag for a product, and GWT says that I have duplicate title tags but there is only one live page for the product. GWT lists the dupe. title tags, but when I click on each they all lead to the same live page. I'm confused, what pages are these other title tags referring to? Does Google have more than one page for that product indexed due to me changing the title tag when the page had a different URL? Does this question make sense? 2) Is this issue a problem? 3) What can I do to fix it? Any help would be greatly appreciated Jeff
Intermediate & Advanced SEO | | IOSC0 -
Navigation - Balancing UX & SEO
I'm currently evaluating our navigation in the course of a site relaunch. From reading a number of articles and posts on seoMOZ, here are the elements I've found important to consider: Use CSS (not Javascript) for the primary drop-down navigation menu Get rid of two design elements from our earlier days: The 30 something site-wide category links in the footer, and many no-followed internal links (in an attempt to sculpt PR) Keep all pages within 3 clicks of the homepage, and have ample cross-links within internal pages. The one major problem I'm facing is how to balance UX and SEO in the primary navigation bar. To illustrate, let's assume I sell Tennis equipment. If one of the top-level categories on my navigation bar was "Rackets", if I was designing purely with SEO in mind the category names would be: Tennis Rackets -> Wilson Tennis Rackets Head Tennis Rackets Prince Tennis Rackets ....as the full, three word anchor text will be most specific and valuable to pass reputation to the category pages. However, from a UX perspective, writing "Tennis Rackets" after each category is unnecessary, and it would look MUCH cleaner to instead have: Tennis Rackets -> Wilson Head Prince ....but this would obviously be less beneficial from a SEO standpoint for each individual, manufacturer racquet page as the entire search term ("Wilson Tennis Rackets") is not in the anchor text. As these links will be on every page of the site, I'm struggling with which to choose - clean navigation or improved SEO. My Questions: I would love to hear the communities thoughts on how to weigh the balance of these two - clean UX navigation vs. SEO-rich specific anchor text - in navigation. Also, I'd appreciate hearing if any of my original 3 assumptions for the re-design are off-base or incorrect. Thank you!
Intermediate & Advanced SEO | | AndrewY0 -
SEO for Global Navigations
I did my first SEO audit from the book SEO Secrets by Danny Dover on my new website at http://melo4.melotec.com:4010/ In the book he says to disable Javascript and see if the global navigation still works. So when I did that the dropdown menus in my navigation don't show. I'm assuming this is a problem but when I check the cache text only version of the site, the dropdowns are in the text only version. Are their any experienced SEO's out their who can weigh in on this issue? Should I have my developer redo the navigation without any javascript? Thanks, Shawn
Intermediate & Advanced SEO | | Romancing0