Robots.txt Question for E-Commerce Sites
-
Hi All,
I have a couple of e-commerce clients and have a question about URLs.
When you perform a search on website all URLs contain a question mark, for example: /filter.aspx?search=blackout
I'm not sure that I want these indexed.
Could I be causing any harm/danger if I add this to the robots.txt file? /*?
Any suggestions welcome!
Gavin
-
You're right on target, it's not a good idea to index search results. Google doesn't want to crawl or index other search results in its own search results. There are some exceptions for gigantico sites like Yelp or TripAdvisor when showing their search results pages are actually the best option, but if you're not at that level and especially if you're an ecommerce site, it's not recommended.
You wouldn't be harming anything by excluding search from your robots.txt file. In fact, many top sites exclude search results to preserve crawl capacity and for indexation reasons.
You'll also want to look at parameter handling in Search Console, this article from Google will get you started.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Changing Url structure to incorporate Woo Commerce
Advise needed please We have rankings coming along nicely with a website that uses page content but we now need to start online shopping with woo commerce The url structure has always been a bit of a mess, but its quite in depth We are looking to move small paragraphs about each product cat (formerly put on Pages) information into the Product Category and then the Product information into the product page and redirect the old urls to the new urls. It would mean updating the permalinks also - My concern if there is less leverage with product categories - do these rank just as well as pages, are we going to see our rankings change dramatically in doing so? Added to that - is it best doing this change gradually or all at once (like staging site to get the set up ready) and then pushing live
On-Page Optimization | | KellyDSD860 -
Site is too slow? Seeing a new code.
Site is too slow. I am seeing this new code more than 1000 times in my home page- What should I do now? My site- http://a1stlucie.com/
On-Page Optimization | | Beachflower0 -
Site folder hierarchy 2016 ?
Hi, Would you say that based only on the "folder structure" that "somedomain.com/focus-keyword" has better conditions to rank well than "otherdomain.com/general-keyword/focus-keyword" ? Using Yoast SEO for WordPress when creating automatic XML sitemaps, URLS with subfolders seems to be getting lower priority. Thanks!
On-Page Optimization | | Agguk0 -
Ecommerce question - overoptimisation
Hi there, I'm the owner of this site http://goo.gl/76Mrbd and I have a newbie question We have a large number of vintage clothing items, and for SEO purposes I would like to rank for keywords "vintage insert designer name". I've been titling all our items with the phrase "vintage insert designer skirt/dress/pants etc" - but I've noticed that when we google these phrases - an example being "vintage moschino" we don't rank for them at all, despite the fact that each individual page gets a decent "page optimisation score" for that particular keyword. Many people have suggested to me that the reason this is so is because we're using the keyword too much on our site, but I can't get my head around how to fix this. What would be the best way to restructure our product descriptions so that we can fix this? Thanks in advance
On-Page Optimization | | clare_f1 -
Amazon powered e-commerce website automatically generates page titles. Is there an alternative?
Greetings fellow Mozzers, So the product pages pick up the product name and convert it into the page title by default. The development company says this is an Amazon limitation and we can't set custom page titles for product pages. Is this the case? The official documentation from Amazon does not seem to suggest so. Would greatly appreciate any feedback from the community. Thanks, Rishad.
On-Page Optimization | | RishadShaikh0 -
Site wide 301 or canonical links.
Hi guys, I'd like add code to my header file to specify www. as opposed to just http:// for the canonical links across my entire site. How can I do this? I'd like it to be site wide code that I can just add to my header.php file which is included across the site.
On-Page Optimization | | absoauto0 -
Can I add a blog to different pages of my site?
I would like to ad a blog to a few different pages of my site, is this possible? I would prefer it to not be a stand alone blog, because I am pretty sure it would begin to outrank my landing pages in search results. I built my site using dreamweaver, but I have seen a reference somewhere to a utility for adding a word press blog to an existing site? This site seems to be doing what I had in mind? http://www.drynclean.com/
On-Page Optimization | | ayetti0 -
Using meta robots 'noindex'
Alright, so I would consider myself a beginner at SEO. I've been doing merchandising and marketing for Ecommerce sites for about a year and a half now and am just now starting to attempt to apply some intermediate SEO techniques to the sites I work on so bear with me. We are currently redoing the homepage of our site and I am evaluating what links to have on it. I don't want to lose precious link juice to pages that don't need it, but there are certain pages that we need to have on the homepage that people just won't search for. My question is would it be a good move to add the meta robots 'noindex' tag to these pages? Is my understanding correct that if the only link on the page is back to the homepage it will pass back the linkjuice? Also, how many homepage links are too many? We have a fairly large ecommerce site with a lot of categories we'd like to feature, but don't want to overdo the homepage. I appreciate any help!
On-Page Optimization | | ClaytonKendall0
