Robots.txt Question for E-Commerce Sites
-
Hi All,
I have a couple of e-commerce clients and have a question about URLs.
When you perform a search on website all URLs contain a question mark, for example: /filter.aspx?search=blackout
I'm not sure that I want these indexed.
Could I be causing any harm/danger if I add this to the robots.txt file? /*?
Any suggestions welcome!
Gavin
-
You're right on target, it's not a good idea to index search results. Google doesn't want to crawl or index other search results in its own search results. There are some exceptions for gigantico sites like Yelp or TripAdvisor when showing their search results pages are actually the best option, but if you're not at that level and especially if you're an ecommerce site, it's not recommended.
You wouldn't be harming anything by excluding search from your robots.txt file. In fact, many top sites exclude search results to preserve crawl capacity and for indexation reasons.
You'll also want to look at parameter handling in Search Console, this article from Google will get you started.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Can access my site using www
Hello, when I try to access my website using www i would like it to redirect to non www but instead it shows a sal error message.
On-Page Optimization | | Voopoo2 -
Moving Site from HTTP to HTTPS
Hi, So the news is that Google has started giving more importance to sites with HTTPS i.e. it is now a new ranking signal. It says that as of now it affects fewer than 1% of global queries, and carrying less weight than other signals such as high quality content but it may decide to strengthen it as they would like to encourage all website owners to switch from HTTP to HTTPS to keep everyone safe on the web. In that case, what should we do? Switching from http:// to https:// means change in urls and low traffic. How to cope with it? Do we have to implement 'n' number of redirects? Regards,
On-Page Optimization | | IM_Learner3 -
Craw structure for web site about jobs?
Hi there, we have now a client who has a job offering web site.There are many craw errors in it. My question is how should the url structure in a jobs website look like and which pages should be indexed? What is the best way and tips for optimizing a job website? Now the posted jobs pages are dynamically like: examplejob.com/detail-job/1891222223/Careers-for-Mens---Womens/Experienced-Web-Design-Need I see many job websites allow their job offers to be indexed and may be this is useful because some people find jobs also when directly search in Google. Are they using dynamically urls for that? And also my related question is what happens when the job offer expires? When Google craws that page again should it be redirected to 404 page or the original job offer text should be there and just to be added info that this job offer has expired? Otherwise If only it's written that it has expired may be there will be too much duplicate content on many many pages.
On-Page Optimization | | vladokan0 -
When it comes to duplicate page content how do I deal with correcting it. Its a dynamic e commerce site.
I am under the impression that with ecommerce sites this happens often and that there's a plug in or just simply not worry about it since queries will often find similar conent.
On-Page Optimization | | Wayne_c0 -
Which pages on my site should I back link to
The majority of the back links I have been creating link directly to our home page and to the store page. Is this the best approach or should I be trying to spread the links throughout our site to include product categories and subcategories etc?
On-Page Optimization | | Hardley0 -
Best Way to Add Google Map to Site
My site is built with WordPress. I want to add our business location in the contact page using Google Map. What is the best way to do this? There is many plugins out there for Word Press do I need to use one ? Hopw about Webmaster tools, does it need to be submit somewhere in there? When it is done properly, will Google show our store location with a map in Goolgle search results? I saw some store result showing the map & business hours, How to do this? Thank you for your help. BigBlaze
On-Page Optimization | | BigBlaze2050 -
Disadvantages & Advantages for this e-commerce URL Structure
Hi all, Just a simple question. What's the disadvantages/advantages to using this URL structure for an ecommerce site: **Category structure: **/category **Subcategory structure: **/subcategory **Product structure: **/product Compared to the standard: /category/subcategory/product Thanks, Nick
On-Page Optimization | | NickPateman810 -
Sister Sites or Joint Family?
A large News Media Group has a Tv Channel, print newspaper, radio channel (for music primarly) and an online website that includes the newspaper content and other original content in different media. My question is, is it better to have independant websites for these different mediums or have all the content on one big website. Currently the newspaper and blog are online as one whereas the radio channel has its own website and the television has its own. So should we maintain sister sites and cross link to each other or have one big happy family under one house? Best, Rishad.
On-Page Optimization | | RishadShaikh590