Can I safely block my product listing from search? Does it even make sense?
-
Hi,
I've an ecommerce website with more than 50k urls and only 10% or so are getting crawled regularly by Google.
Product listing pages represent roughly 80% of these 50k pages.Trying to improve this, I was thinking to remove altogether all (most?) of my product listing from search (via Robot.txt) to keep only the product pages themselves and the product categories.
My organic situation since Jan 2019:
Users: 2,300,000 (of which 9% are visiting product listing pages)
Page views: 8,000,000 (of which 5% are product listing pages).Am I about to unleash armageddon (or more like harakiri) on my website by doing so or actually get Google to crawl much more relevant resources (product pages, product categories, blog content and so on)?
Thanks,
G -
Have had a lot of success with that kind of deeper logic in the past, you can usually quite easily create such rules using robots.txt wildcards
-
Thanks for your answer and the link, that's actually very useful.
What I mainly struggle with is to understand what I can prune/not and based on what criterias. And the article you've linked is helping a fair bit on that aspect.I'm thinking to start on multiple filter pages with a rule such as "block product listing pages from being indexed if at least 2 filters have been selected". And then see how it impacts the site and my crawl budget.
-
Hello GhillC,
I think we need to agree on terminology first, but it sounds like you can safely limit some of Google's access. Some people call your "product listing pages" either "refinements", "facets", or "filters". When I read "product listing pages" I typically think of what is also called a "category" page, which is a page listing multiple products. A single product page is often referred to as a product detail page (PDP).
Now that we're on the same page (pun intended), let me know if this article answers your question. It is very dated (2011) but gets the point across, which is that you need to be strategic about which facets/refinements/filters you allow to be crawled and/or indexed: https://moz.com/blog/building-faceted-navigation-that-doesnt-suck .
One more thing: 5% - 9% of traffic going directly from organic search into a page type would be considered significant for most businesses. When I look at pruning out page types, they're typically responsible for less than 0.5% of traffic from organic search.
-
A product page is a page speaking about a selected product.
Example: a page about the smartphone Samguns Supernova 30A product category is a page speaking about the category, or the type of product it falls into.
Example: smartphone or, if it is too broad, the Supernova lineA product listing is a page listing all the product refined with available criterias such as a product line.
Example: a page listing all the Samguns Supernova phones
Or a page listing all the Samguns Supernova phones with more than 128GB of HD.
Etc.Does it make more sense?
-
Can you explain the difference between the 'products listings' and the 'actual products themselves'?
You say you still want products and product categories to rank, but not product listings. But to most readers, a product listing is usually a product category or product page (so the info seems to contradict itself, which actually it may not do - just needs more explaining)
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Australian search - ZERO visibility and stumped
Fair warning, this is going to be long, but necessary to explain the situation and what has been done. I will take ANY suggestions, even if I have tried them already. We have a sister site in Australia, targeting Australian traffic. I have inherited what seems to be an incredible rat's nest. I've fixed over two dozen issues, but still haven't seemed to address the root cause. NOTE: Core landing pages have weak keyword targeting. I don't expect much here until I fix this. The main issues I'm trying to resolve first are with the unusual US-based targeting, and the inability of the homepage to rank for anything. The site is www[dot]castleford[dot]com[dot]au. Here's the rundown on what's going on: Problems: The site ranks for four times as many keywords in the US as it does in Australia. The site ranks for a grand total of 5 keywords on the first page for AU keywords. The homepage, while technically optimized on-page for "content marketing agency", and with content through MarketMuse, has historically ranked between 60-100, despite having a fairly strong DA with fairly weak competitors, based on AHREFs keyword difficulty, and Moz keyword difficulty. Oddly, the ranking has gone up to 5-7 for three day spurts over the past year. Infrequent indexing of homepage (used to be every 2-3 weeks, I've gotten that down to 1 week). Sequence of events: November 2017 - they made some changes to their URLs - some on the blog and some on the top nav LPs. Redirects seem okay. November 2017 - Substantial number of lost referring domains, not many seem to be quality. January 2018 - total number of AU ranking keywords more than halved. May/June 2018 - added a follow inbound link sitewide to an external site that they created. 20k inbound links with same anchor text to homepage. Site has a total of 24k inbound links. July-Sep 2018 - total number of US ranking keywords halved November 10 - I walked into this mess. What's been done: Reduced site load speed by over 150% (it was around 20 seconds). Create sitemap (100 entry batching) and submit to GSC. Improved MarketMuse score for the homepage. Changed language from "en-US" to "en-AU" Fetch and render - content is all crawlable and indexed properly. Changed site architecture for top nav core landing pages to establish clear hierarchy. All version of GSC created, non-www and www http, and non www https and www https Site crawl - normal amount of 404s, nothing stands out as substantial. http to https redirect okay. Robots.txt updated and okay. Checked GSC international targeting, confirmed AU. No manual links penalty I'm clearly stumped and could use some insights. Thanks to everyone in advance, if you can find time.
Technical SEO | | Brafton-Marketing0 -
How to block text on a page to be indexed?
I would like to block the spider indexing a block of text inside a page , however I do not want to block the whole page with, for example , a noindex tag. I have tried already with a tag like this : chocolate pudding chocolate pudding However this is not working for my case, a travel related website. thanks in advance for your support. Best regards Gianluca
Technical SEO | | CharmingGuy0 -
PhantomJS to Make AJAX Pages Crawlable
Anyone have any experience using PhantomJS to return HTML snapshots of AJAX rendered pages? More specifically, does anyone know if Google takes issue with this technique in any way? Interested in learning about this technique? Using PhantomJS to allow Googlebot to crawl your AJAX pages.
Technical SEO | | RyanOD0 -
Increase Search Ranking for CEO
Hi guys My company CEO is concerned that when her name is googled pictures of a glamour model appear in the image results area. The glamour model shares a second name with our CEO and this is why the model's images are appearing. I have been asked to rectify this situation. My CEO has a linked in page and twitter account which are underused but no personal page on our company website. I was thinking of buying the url for the CEO's name and optimizing a small site for her name with bio etc and links to twitter, lined in etc. Would this be the best strategy? Thanks Gavin
Technical SEO | | gavinr0 -
Sitemap.xml showing up in Google Search
Hello when I do a Google search my sitemap.xml shows up for lots of queries. Does anyone have any advise on this? Should I remove url in Google Webmaster? Thanks,
Technical SEO | | Socialdude0 -
Competition links make no sense
Hello everybody, I used the open site explorer to check where my competitor has links and try to put mine there too. However I am extremely confused with the results. Eg the first link to my competitor coming from a domain with authority 91, is a download file. The other one is a link from ups, the courier service. When I click on it I get an access denied.The other one comes from samsung and when I click on it, I download an swf file. Next one, fcc.gov and it downloads a wp file. If I keep clicking on these links, in the end I am going to get a virus or something and learn nothing about what my competitor does. Any one have a clue how they managed to get linked like that?
Technical SEO | | polyniki0 -
Tool for extracting search queries
Hello, Does anyone know of or have a tool that takes referrer URLs coming from Google which extracts the search query from the URL string? Thank you
Technical SEO | | soeren.hofmayer0 -
Search Engine Blocked by Robot Txt warnings for Filter Search result pages--Why?
Hi, We're getting 'Yellow' Search Engine Blocked by Robot Txt warnings for URLS that are in effect product search filter result pages (see link below) on our Magento ecommerce shop. Our Robot txt file to my mind is correctly set up i.e. we would not want Google to index these pages. So why does SeoMoz flag this type of page as a warning? Is there any implication for our ranking? Is there anything we need to do about this? Thanks. Here is an example url that SEOMOZ thinks that the search engines can't see. http://www.site.com/audio-books/audio-books-in-english?audiobook_genre=132 Below are the current entries for the robot.txt file. User-agent: Googlebot
Technical SEO | | languedoc
Disallow: /index.php/
Disallow: /?
Disallow: /.js$
Disallow: /.css$
Disallow: /checkout/
Disallow: /tag/
Disallow: /catalogsearch/
Disallow: /review/
Disallow: /app/
Disallow: /downloader/
Disallow: /js/
Disallow: /lib/
Disallow: /media/
Disallow: /.php$
Disallow: /pkginfo/
Disallow: /report/
Disallow: /skin/
Disallow: /utm
Disallow: /var/
Disallow: /catalog/
Disallow: /customer/
Sitemap:0