Robots.txt Question for E-Commerce Sites
-
Hi All,
I have a couple of e-commerce clients and have a question about URLs.
When you perform a search on website all URLs contain a question mark, for example: /filter.aspx?search=blackout
I'm not sure that I want these indexed.
Could I be causing any harm/danger if I add this to the robots.txt file? /*?
Any suggestions welcome!
Gavin
-
You're right on target, it's not a good idea to index search results. Google doesn't want to crawl or index other search results in its own search results. There are some exceptions for gigantico sites like Yelp or TripAdvisor when showing their search results pages are actually the best option, but if you're not at that level and especially if you're an ecommerce site, it's not recommended.
You wouldn't be harming anything by excluding search from your robots.txt file. In fact, many top sites exclude search results to preserve crawl capacity and for indexation reasons.
You'll also want to look at parameter handling in Search Console, this article from Google will get you started.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
For an e-commerce product category page that has several funnels to specific products, for SEO purposes does it matter whether the category page's overview content is above or below those funnels?
We manage an e-commerce site. On a category page, there are several funnels to specific products. We moved the category overview content below those funnels to make it easier for users to quickly get to products. Seems more user friendly to me, but could that move of the main content to the lower part of the page be a negative ranking factor?
On-Page Optimization | | PKI_Niles0 -
Need suggestion: Should the user profile link be disallowed in robots.txt
I maintain a myBB based forum here. The user profile links look something like this http://www.learnqtp.com/forums/User-Ankur Now in my GWT, I can see many 404 errors for user profile links. This is primarily because we have tight control over spam and auto-profiles generated by bots. Either our moderators or our spam control software delete such spammy member profiles on a periodic basis but by then Google indexes those profiles. I am wondering, would it be a good idea to disallow User profiles links using robots.txt? Something like Disallow: /forums/User-*
On-Page Optimization | | AnkurJ0 -
Site wide 301 or canonical links.
Hi guys, I'd like add code to my header file to specify www. as opposed to just http:// for the canonical links across my entire site. How can I do this? I'd like it to be site wide code that I can just add to my header.php file which is included across the site.
On-Page Optimization | | absoauto0 -
Best Way to Add Google Map to Site
My site is built with WordPress. I want to add our business location in the contact page using Google Map. What is the best way to do this? There is many plugins out there for Word Press do I need to use one ? Hopw about Webmaster tools, does it need to be submit somewhere in there? When it is done properly, will Google show our store location with a map in Goolgle search results? I saw some store result showing the map & business hours, How to do this? Thank you for your help. BigBlaze
On-Page Optimization | | BigBlaze2050 -
Do you handle website on-page/site optimization?
Our website is gaining traction and we are looking for an individual who is an expert at examining websites to see if the structure and on-page optimization is being optimized, and what. Please contact me if you are interested. Thank you.
On-Page Optimization | | balboafinance0 -
Site Redesign: Potential Problems?
We're looking at a cleanup of our site and making our navigation menu simpler and more visual. The problem is there's a massive fear that we will incur some sort of SEO hit at a time when we can ill afford one. The bulk of the changes are around some tertiary pages that have some SEO value but are not primary targets or traffic sources. The pages with content aren't moving URLs or changing, nor is the core content of the site, just the template. The new template, I feel, is easier to navigate and easier to spider (menus are flyout divs with images). So are there any tips or tricks to changing the layout of your site without hurting SEO? Anything we need to avoid?
On-Page Optimization | | Highland0 -
Why is our site dropping in Domain Authority
Hi all, I have a question. I have been working SEO on our site www.betxpert.com for the last 4-6 weeks and I have been looking forward to seeing the new DA and PA numbers. However today, our DA is now 30...where it was 31 or 33 a couple of days ago. How come the decrease? I have only improved the site i should say... Hope someone can shed a little light on the issue... -Rasmus
On-Page Optimization | | rasmusbang0 -
Site-wide keyword density
A colleague of mine was saying that he has been able to get top ranking for a high traffic term by using variations of that head term on multiple pages that are associated with the main page. For example,he would optimize a landing page for the high traffic word "Construction." He would then build pages under this landing page that are optimized for variations of this word: "Construction facts," "Industrial Construction Companies," "Construction Resource Allocator" etc. His theory is that the subpages add credibility with spiders that the root page is the best for that root page. This doesn't seem like it would work, but I'm curious as to what other people think.
On-Page Optimization | | EricVallee340