Robots.txt Question for E-Commerce Sites
-
Hi All,
I have a couple of e-commerce clients and have a question about URLs.
When you perform a search on website all URLs contain a question mark, for example: /filter.aspx?search=blackout
I'm not sure that I want these indexed.
Could I be causing any harm/danger if I add this to the robots.txt file? /*?
Any suggestions welcome!
Gavin
-
You're right on target, it's not a good idea to index search results. Google doesn't want to crawl or index other search results in its own search results. There are some exceptions for gigantico sites like Yelp or TripAdvisor when showing their search results pages are actually the best option, but if you're not at that level and especially if you're an ecommerce site, it's not recommended.
You wouldn't be harming anything by excluding search from your robots.txt file. In fact, many top sites exclude search results to preserve crawl capacity and for indexation reasons.
You'll also want to look at parameter handling in Search Console, this article from Google will get you started.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Best way for navigation, niche E-commerce
Hi community friends. Ive have started to work on our 2.0 e-commerce store mostly when it comes to design and the ux. Its a woocommerce setup and in total we have 34 products! We have e niche site where we sell products as a system towards the building industry. Each product served its purpose within a construction detail and needs some explanation for a first time user/customer. So, first thing we did now was to ad the short description to our loop to bring some more info to the user and guide them in the right direction quick. Now the products are divided in to four categories. We actually gain a lot of traffic from those category landing pages and i am not so eager to change the way the products are categorised. The old menu where built up by a sidebar with links to the different categories and to tags pages. Where the category where for example "Adhesive" and the Tag "Seal a window from the outside". The category gives the broader direction, and the tag gives the final usage/solution/purpose with the product. In the 2.0 we would like to remove the sidebar in the product pages in favor for a full widht product page with more focus on the product and the specs. This leads us to put our navigation up in the main menu part for the desktop version. Now to the questions. 1. Do you think that we are doing it the right way, to niche and set the final usage/path to our products with tags, or should it be done in another way? 2. For the UX, do you think that removing the sidebar navigation on the product pages will lead to a worse user/shopping experience? Attached you can see some pics from our dev server where the new menu are being built up, and also the new product page layout. Please give me some feedback! 🙂 // J open?id=0BzS3SfsD0lmcSFR4a21rRlVBcjA open?id=0BzS3SfsD0lmcM0djUi1pcXdrOWM
On-Page Optimization | | knubbz0 -
How many css and Javascript on ecommerce site?
Hello, I want any tool which seach css and javascript of whole ecommerce cite? Please suggest. Thanks! Dev
On-Page Optimization | | devdan0 -
Site restructure question
Our site was deigned years ago to target customers in specific cities, now we've grown beyond this and I believe it is time to change the site structure.
On-Page Optimization | | PM_Academy
Ignore the 302 from the root page. Current structure: (assuming you've never been to our site before) projectmanagementacademy.net 302->/select-location.php /select-location.php -> /city-name/pmp-training.php This page was meant to be a "homepage" for each city, pointless page really /city-name/pmp-training.php -> /ciy-name/product-name.php These pages are for each individual product My suggested site structure: /city-name/pmp-training.php becomes projectmanagementacademy.net no more redirect /city-name/pmp-training.php gets removed and 301 to root page. /product-name.php each product's page and you would select a location when necessary (some products are online only) would 301 each /city-name/product-name to corresponding product page /product-name/city-name.php could add these pages if we still wanted the city name in url for city specific products My thoughts here are /product-name.php would receive a higher % of link juice because there are fewer page between 2 vs 4 if you came to the root page. and 2 vs 3 if you came from the select-location page. Also instead of being split between over 50 locations, all these would be together on one page. Your thoughts? Would this change improve our SERP for those product pages? Would we see a drop off in traffic if we did this? How long, if done correctly, would it take to see the recovery of rankings and traffic? Could we 301 /select-location.php to the root page? Thanks in advance for your insights to this. Any answer is a good answer. Trenton0 -
When it comes to duplicate page content how do I deal with correcting it. Its a dynamic e commerce site.
I am under the impression that with ecommerce sites this happens often and that there's a plug in or just simply not worry about it since queries will often find similar conent.
On-Page Optimization | | Wayne_c0 -
Scripts question
My question is whether search engines will crawl the page before or after scripts run on the page. I have a H1 that will be replaced with a canvas by jquery script. Will search engines see the H1 or the canvas?
On-Page Optimization | | lojdqvist0 -
Website redesign: site going from .php to .html
A site I'm working on is being redesigned because the current platform does not allow for content to be changed easily. In the process, they are going from .php to .html. I am concerned about their losing link juice. Can a site work with the old content remaining .php and the new content being .html or should all pages stay .php?
On-Page Optimization | | cakelady0 -
Robots.txt file
Does it serve any purpose if we omit robots.txt file ? I wonder if spider has to read all the pages, why do we insert robots.txt file ?
On-Page Optimization | | seoug_20050
