Robots.txt Question for E-Commerce Sites
-
Hi All,
I have a couple of e-commerce clients and have a question about URLs.
When you perform a search on website all URLs contain a question mark, for example: /filter.aspx?search=blackout
I'm not sure that I want these indexed.
Could I be causing any harm/danger if I add this to the robots.txt file? /*?
Any suggestions welcome!
Gavin
-
You're right on target, it's not a good idea to index search results. Google doesn't want to crawl or index other search results in its own search results. There are some exceptions for gigantico sites like Yelp or TripAdvisor when showing their search results pages are actually the best option, but if you're not at that level and especially if you're an ecommerce site, it's not recommended.
You wouldn't be harming anything by excluding search from your robots.txt file. In fact, many top sites exclude search results to preserve crawl capacity and for indexation reasons.
You'll also want to look at parameter handling in Search Console, this article from Google will get you started.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Ecommerce Canonical Question
Hi all, first question (eek) Could I pick the brains of fellow users around an issue we are having with canonical urls on a magento website. At the moment we do not have these enabled as it seems to break our indexing. Cut a long story short, we have thousands of products but haven't rewritten many of the descriptions from the manufacturers yet and so have noindexed all the product pages (freeing them as we go). The goal, for now, is to pull in traffic via the filtering options we have on the site The goal, for now, is to pull in traffic via the filtering options we have on the site. For example, if you go to Dresses, there then are several filtering options which would allow you to choose a colour, shape and material - if you wished to filter that precisely. These filtering options are all crawlable and so we would then have a page that google could index for, for example, Green Lace Maxi Dress. All good there, few people search for specific products and a lot search for types of products so we are covered. To get back to the issue at hand. If we enable the canonical option on our magento plugin it will stop us from being able to target these terms. Whereas the filtering option would create domain.com/dress/green/maxi/lace with the page title of Green Lace Maxi Dress, if we enable the canonical part of the seo plugin the canonical link which would be added to the page would be - instantly removing our ability to rank for longer tail dress related searches (we are not going to compete with the big players on the premium terms, yet!). There are alternative plugins we can buy for magento to add the correct tag, however, if every page's canonical just points back it itself like this, is there really much point spending nearly $1000 on the 4 licences we would need to cover our range of sites. Is it really necessary, in this case, that we have a canonical for the product filtering? Sorry for the long post, hope it made sense. Thanks for any assistance.
On-Page Optimization | | DSCarl0 -
Canonical Tag for Ecommerce Site
My client has an ecommerce site with over 1,000 products. We have a ton of duplicates because of how their ecommerce system handles product pages. Each time a new product is added, there is a default product page created (/product/12345-product-name.aspx). Each time that product is added to a specific product category, another, separate URL is created (/product/office-chairs/12345-product-name.aspx). The site has over 1,000 duplicates (at least one for each product) because of how the ecommerce system structures URLs. We are unable to have unique content on /products/12345-product-name.aspx and /product/office-chairs/12345-product-name.aspx because both pages pull from the same database. Their webteam informed me that they can't implement canonical tags on individual pages, they must be dynamically added to the site all at once. Thus forcing me to choose all of the default product pages as primary URLs. Both types of URLs are getting indexed and the product URLs that were added to the categories are SEO friendly so I'm leary to eliminate one or the other with a canonical tag or a no index. Suggestions?
On-Page Optimization | | DynoSaur0 -
New website launch - Dupilicate content question
Hi, We are going to be launching a US version of our UK E-commerce web store. The US version will be a standalone site, separate URL/hosting etc. My question is that the US website will carry the same products as the UK site - will the product copy have to be written differently for the US website in order to avoid any SEO duplicate copy? Thanks
On-Page Optimization | | WalesDragon0 -
Best information organization for a new site?
I'm launching a new stain removal website, and wanted to know what would be considered the best way to organize the content? Since most articles will roughly involve "removing X from Y" or "how to remove Z," I can see two ways... 1. Organize articles by Stained Items, Stain Agents and perhaps Cleaning Detergents. 2. Spread the categories out more, to try and group stained items according to categories... E.g. Hard surfaces, delicates, fabrics, ceramics etc. Any thoughts on which of these two might be the best way to organize the site, or are there any better suggestions? Not sure what the main considerations are here... Either of these two seem equally user-friendly.
On-Page Optimization | | ZakGottlieb710 -
I run a Q&A site, should I limit post questions to 64 characters?
I run a Q&A site (Recessionitis.com), should I limit posts (questions) to 64 characters for SEO purposes?
On-Page Optimization | | 10JQKAs0 -
Too many on page links on ecommerce site
I have an online store with 10 catagories, many of those have subcategories. I have a tree style navigation menu on the page. This helps people quickly find what they need. However, I end up with about 125 links on the page that way. Does google really penalize me for this? Is there anyway around this? Advice much appreciated!
On-Page Optimization | | bhsiao0 -
URL structure for a new WordPress site
Hi I'm building a new next big thing website from scratch (for a translation agency) and I encountered an issue with the URL structure. I need to chose the URL for important targeted keyword pages and I have a conflict between two tools I'm using. Please read below the situation: domain: mashtranslation.com target keyword: french translation services which URL you think is better from a SEO point of view (and possibly for users): mashtranslation.com/services/french/ OR mashtranslation.com/french-translation-services/ I'm asking this because one WordPress plugin (Wordpress SEO by Yoast) says the URL structure is not optimised while another tool (Market Samurai) says the URL is optimised.
On-Page Optimization | | flo20 -
Duplicate Content Question
On the home page of my site I have a read more link that takes you to a different URL with basically the same content, just more of it. Home Page: http://www.opwdecks.com/ Read More Link on Home Page: http://www.opwdecks.com/deckmaintain.htm I think this may be affecting my seo. Any suggestions on what I should do about this? Should I add a canonical to the home page and/or on the other page? Both pages are indexed by google. Thanks for any help or tips.
On-Page Optimization | | opwdecks0