Robots.txt Question for E-Commerce Sites
-
Hi All,
I have a couple of e-commerce clients and have a question about URLs.
When you perform a search on website all URLs contain a question mark, for example: /filter.aspx?search=blackout
I'm not sure that I want these indexed.
Could I be causing any harm/danger if I add this to the robots.txt file? /*?
Any suggestions welcome!
Gavin
-
You're right on target, it's not a good idea to index search results. Google doesn't want to crawl or index other search results in its own search results. There are some exceptions for gigantico sites like Yelp or TripAdvisor when showing their search results pages are actually the best option, but if you're not at that level and especially if you're an ecommerce site, it's not recommended.
You wouldn't be harming anything by excluding search from your robots.txt file. In fact, many top sites exclude search results to preserve crawl capacity and for indexation reasons.
You'll also want to look at parameter handling in Search Console, this article from Google will get you started.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
I have a site with jokes. What schema markup could I use?
My site is about jokes. I wonder what schema markup could I use to be more visible in the search results.
On-Page Optimization | | MichaelJanik0 -
Translated the site but traffic is not coming
Hi, We've build a lawyer directory website (www.iranianlawyers.com) which already has good Google rankings for related terms (Ex. Iranian lawyers, Iranian lawyers california, etc). About 1.5 months ago we translated the site to Farsi and published it online: www.iranianlawyers.com/farsi However we have yet to see any new traffic generated from those pages. The website has a decent back-link profile and there are no almost no competitors in our space with translated pages. Would someone please take a look at our translated pages and let me know if there are any major on-site issues that you see that we need to address? I've checked for noindex or nofollow tags but they dont seem to be an issue. Not sure if I'm missing something here. Thank you very much
On-Page Optimization | | Heydarian0 -
Two Robots.txt files
Hi there Can somebody please help me that one of my client site have two robot.txt files (please see below). One txt file is blocked few folders and another one is blocked completely all the Search engines. Our tech team telling that due to some technical reasons they using second one which placed in inside the server and search engines unable to see this file. www.example.co.uk/robots.txt - Blocked few folderswww.example.co.uk/Robots.txt - Blocked all Search Engines I hope someone can give me the help I need in this one. Thanks in advance! Cheers,
On-Page Optimization | | TrulyTravel
Satla0 -
HTML Site SEO (NO CMS)
I have got a client site, which is dated (2007) and has not been shifted to any recognised CMS yet. It is HTML based. Is it possible to SEO on such a site? Is it even worth it? If it is possible to do SEO on this, any suggestions will be highly appreciated. Thank you.
On-Page Optimization | | ArthurRadtke3 -
Looking for an e-commerce CMS that can deal properly with facets and filters
Hi! One of my clients has an online shop built on a custom platform. Following my on-page audit, I found multiple issues with the faceted & filtered navigation. After doing some research, I've developed a good site architecture based both on keyword targeting and on user experience. Since the client's platform is custom and he is no longer in contact with the original webmaster, I'm looking to recommend an e-commerce CMS that would fix the issues he has and allow for proper navigation using facets + filters. I'll give you an idea of how I've envisioned the new site architecture (with a random example): The website has 3 main categories: Jackets Shoes Pants These categories are completely separated, no products exist in more than one category. Let's take the Shoes category. It has the following facets (these are like subcategories, in the form of links, indexable by the search engines): By Brand: Brand 1 Brand 2 Brand 3 By Genre Men Women Unisex By Type Running Flip-Flops Boots Each product of the Shoes category will belong to one of each facets (so you can have a product that belongs to Brand 1, for Women, of the "boots" type). Now, each of these facets also display filters depending on the facet itself. These are non-indexable, and preferably done through AJAX. For example, if I select Shoes -> Men I get a list of filters that can also include options from the other facets (so I'll have filters for "Size", but also for "Brand" and "Type"). When I select Shoes -> Boots I won't have the "Type" filter anymore (or I can have it pre-selected I guess), but I will have the "Size" and "Brand" filters. So, what I am looking for is for a CMS that can allow product attributes (like "Type", "Genre" or "Brand") to be used both as facet categories as well as filters. The closest live example I could find is ASOS, where you can search products by Brand (facet) and have the refined afterwards by Type (filter) or you can search products by Type (facet) and refine them by Brand (filter). So, any idea of an e-commerce CMS that would allow me to do the above (bonus points if it also allows entering descriptions for facets, similar to what ASOS does)? Thanks!
On-Page Optimization | | mihaiaperghis0 -
Need suggestion: Should the user profile link be disallowed in robots.txt
I maintain a myBB based forum here. The user profile links look something like this http://www.learnqtp.com/forums/User-Ankur Now in my GWT, I can see many 404 errors for user profile links. This is primarily because we have tight control over spam and auto-profiles generated by bots. Either our moderators or our spam control software delete such spammy member profiles on a periodic basis but by then Google indexes those profiles. I am wondering, would it be a good idea to disallow User profiles links using robots.txt? Something like Disallow: /forums/User-*
On-Page Optimization | | AnkurJ0 -
Meta Description Question
Hi, Lets say you have a 1000 pages, is it ok to generate a template meta description and flip the keywords depending on the page? For example: Red Shoes page: Buy Red Shoes at amazing prices. We have a large range of Red Shoe sizes available. Blue Shoes page: Buy Blue Shoes at amazing prices. We have a large range of Blue Shoe sizes available. Thoughts?
On-Page Optimization | | Bondara0 -
E-commerce product and review pages
Hi, Can somebody tell me what is the best solution for optimizing review pages of product for e-commerce site? For now situation is this: When somebody write review for product, url-s automatically generate for review page. Review page has same page title, meta description (which is the product description) as product page and link with anchor (excact product name) point back to orginal product. The problem is that I often see in SERP that actually review pages rank better than original product pages. Regards, Nenad
On-Page Optimization | | Uniline0