Robots.txt Question for E-Commerce Sites
-
Hi All,
I have a couple of e-commerce clients and have a question about URLs.
When you perform a search on website all URLs contain a question mark, for example: /filter.aspx?search=blackout
I'm not sure that I want these indexed.
Could I be causing any harm/danger if I add this to the robots.txt file? /*?
Any suggestions welcome!
Gavin
-
You're right on target, it's not a good idea to index search results. Google doesn't want to crawl or index other search results in its own search results. There are some exceptions for gigantico sites like Yelp or TripAdvisor when showing their search results pages are actually the best option, but if you're not at that level and especially if you're an ecommerce site, it's not recommended.
You wouldn't be harming anything by excluding search from your robots.txt file. In fact, many top sites exclude search results to preserve crawl capacity and for indexation reasons.
You'll also want to look at parameter handling in Search Console, this article from Google will get you started.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Correct robots.txt for WordPress
Hi. So I recently launched a website on WordPress (1 main page and 5 internal pages). The main page got indexed right off the bat, while other pages seem to be blocked by robots.txt. Would you please look at my robots file and tell me what‘s wrong? I wanted to block the contact page, plugin elements, users’ comments (I got a discussion space on every page of my website) and website search section (to prevent duplicate pages from appearing in google search results). Looks like one of the lines is blocking every page after ”/“ from indexing, even though everything seems right. Thank you so much. FzSQkqB.jpg
On-Page Optimization | | AslanBarselinov1 -
Include Site Name in Page Titles or not
i would like to ask if it is a good practice or not to Include Site Name in Page Titles. My page is not selling products it is about plagiarism checker tool. i will give one example in one page we are writing about the plagiarism types so the page title is plagiarism types and then is the site name. what is the better practice? Keep it or not? thanks in advance
On-Page Optimization | | anavasis3 -
Duplicate content question
Hi I have a site that is run off one CMS system but has 3 different web addresses. One is a comic shop, one is a toy shop and one is a game shop. Now due to the nature of what we are selling some of the products we are selling on both or all 3 of the sites. I was wondering as to whether this would affect my ability to rank in google and if i would be penalised for any duplicate content? Thanks in advance
On-Page Optimization | | TheZenAgency0 -
Should I remove the Jetpack Plugin From A SIte
I dont know if anyone has any experience with the jetpack plugin, but personally I prefer yoast. My point is someones site I am looking at has both Yoast SEO plugin and also Jetpack for wordpress, should I just remove the jetpack as it seems to be a very heavy loading plugin.
On-Page Optimization | | propertyhunter0 -
Mentioning own site and keywords on here?
I have noticed that sometimes posters will talk about a site without mentioning what it is. I assume this is because it one of their clients so there is confidentiality, is there any other reason I should be aware of? its just that as I am new I am usually cautious and am considering posting my own site and mentioning all my keywords to ask for people’s verdict for my on-page SEO. Still working on it, will be ready soon, thought I would ask in advance. Regards,
On-Page Optimization | | Zoolander0 -
Wordpress permlinks SEO question
What are SEO advantages or disadvantages of the following - /%postname%/ - adds a trailing / /%postname% - no trailing / /%postname%.html - creates .html URL Thanks
On-Page Optimization | | SamBuck0 -
How do we handle sitemaps in robots.txt when multiple domains point to same physical location?
we have www.mysite.net, www.mysite.se, www.mysite.fi and so on. all of these domains point to the same physical location on our webserver, and we replace texts given back to client depending on which domain he/she requested. My problem is this: How do i configure sitemaps in robots.txt when robots.txt is used by multiple domains? If I for instance put the rows Sitemap: http://www.mysite.net/sitemapNet.xml
On-Page Optimization | | nordicnetproducts
Sitemap: http://www.mysite.net/sitemapSe.xml in robots.txt, would that result in some cross submission error?0
