Robots.txt Question for E-Commerce Sites
-
Hi All,
I have a couple of e-commerce clients and have a question about URLs.
When you perform a search on website all URLs contain a question mark, for example: /filter.aspx?search=blackout
I'm not sure that I want these indexed.
Could I be causing any harm/danger if I add this to the robots.txt file? /*?
Any suggestions welcome!
Gavin
-
You're right on target, it's not a good idea to index search results. Google doesn't want to crawl or index other search results in its own search results. There are some exceptions for gigantico sites like Yelp or TripAdvisor when showing their search results pages are actually the best option, but if you're not at that level and especially if you're an ecommerce site, it's not recommended.
You wouldn't be harming anything by excluding search from your robots.txt file. In fact, many top sites exclude search results to preserve crawl capacity and for indexation reasons.
You'll also want to look at parameter handling in Search Console, this article from Google will get you started.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Site not showing up in Google search since move
Hi, hoping someone might help me with some answer(s) as to why our site no longer shows up in Google search results. Even when we type the full name and city into Google, the site is absent. Our Facebook page, LinkedIn and some backlinks show, but our site is missing. I can no longer find it in Google places. I'm sure I've done something wrong since moving from a static (flash-based) site to Wordpress. But the robots.txt file looks okay to me and the sitemap.xml file is present. Anyway, this is what happens when you ask a network technician handle website design... We know just enough to be dangerous! Here is the site in question: www.newfrontiertechnologies.com located in Shoreline WA. Any advice is much appreciated.
On-Page Optimization | | NFTECH0 -
Site wide 301 or canonical links.
Hi guys, I'd like add code to my header file to specify www. as opposed to just http:// for the canonical links across my entire site. How can I do this? I'd like it to be site wide code that I can just add to my header.php file which is included across the site.
On-Page Optimization | | absoauto0 -
Do Blog Comments On Your Site Help SEO?
There is a lot of debate as whether or not having comments on your blog is helpful from an SEO perspective. Proponents believe that more comments (1) creates more content, which search engines love, (2) creates more relevant keywords that can be searched, and (3) helps with "freshness" of the site/content leading to greater site authority. Others like Joost de Valk believe that comments can actually hurt SEO because keyword density cannot be controlled. He argues that his top SEO content are pages not posts for this very reason. What is your opinion?
On-Page Optimization | | marcperry0 -
Moving a site as a folder of existing one
We have the following case: Site A with domain authority 65 and 2,750 linking root domains (139K total links) and Site B with domain authority 68 and 1,336 linking root domains (38K total links) There's a suggestion to move Site A as a folder of Site B so it becomes something like this: domainB.com/domainA using 301 redirects for the existing domain A. We'd like to better understand what such move will result in. I imagine at first Site A will drop in rankings, but after that will it be better for it to be under Site B domain? Also, moving all the pages a step behind in the URL path may slightly affect their rankings. What do you think? Would you move Site A or leave it as a separate domain? I understand there can't be an exact estimation what will happen, but I'll appreciate your thoughts.
On-Page Optimization | | lgrozeva0 -
Can embedding videos from other sites help SEO?
I work for an eCommerce site and many of the vendors we get our products from have videos our their site that explain how the products work and the different features of them, ect. We really have done nothing with video marketing, and would like to create some of our own unique videos, but this will be impossible for the vast majority of the products on our site. We know from a content standpoint embedding videos from our vendors to our product pages will be beneficial, but at the same time it shows our competitors and other site visitors where we are getting our products from which we would rather not do. How will this effect us from an SEO standpoint though? Will it help us to have new content, even though it is embedded from another site? Or will it hurt us because it is not unique? Or will it have no effect at all? Thanks for you help!
On-Page Optimization | | ClaytonKendall0 -
How to make google not index quotes from other sites?
Hey guys, I have a site where we post quite a lot of info from other sites. We don't want google to de-index our pages because parts of it are quotes from other sites. What would you use to make it so Google sees it's a quote from another site? Or to just make Google not index the quote? Thanks!
On-Page Optimization | | StefanJDorresteijn0 -
What is wrong with this site
Hello, I need advice regarding design ( current design as well as any changes ) and seo advice as to what we should do to promote the site. The site in question is - Allkindofessays.com Thanks
On-Page Optimization | | seoug_20050 -
Site Speed Expert Referral Needed
We're looking for someone who can diagnose some site speed issues AND adjust the code, servers, etc. as needed. Does anyone know of an expert in this area?
On-Page Optimization | | BryanPhelps-BigLeapWeb0