Robots.txt Question for E-Commerce Sites
-
Hi All,
I have a couple of e-commerce clients and have a question about URLs.
When you perform a search on website all URLs contain a question mark, for example: /filter.aspx?search=blackout
I'm not sure that I want these indexed.
Could I be causing any harm/danger if I add this to the robots.txt file? /*?
Any suggestions welcome!
Gavin
-
You're right on target, it's not a good idea to index search results. Google doesn't want to crawl or index other search results in its own search results. There are some exceptions for gigantico sites like Yelp or TripAdvisor when showing their search results pages are actually the best option, but if you're not at that level and especially if you're an ecommerce site, it's not recommended.
You wouldn't be harming anything by excluding search from your robots.txt file. In fact, many top sites exclude search results to preserve crawl capacity and for indexation reasons.
You'll also want to look at parameter handling in Search Console, this article from Google will get you started.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Image Titles and Descriptions Question
Hello, I have a question about optimizing the SEO on my pages through image titles and descriptions. There are a few times on my website that I use the same image on multiple pages. I am under the impression that giving it a title such as "social-media-marketing-agency-graphic.jpg" will help the SEO for the phrase "social media marketing agency" on that page. My question was, if I want to use the same image on multiple pages, am I better off uploading an entirely new image with a new title to make it more relevant to the new page? Or will this not make large enough of a difference? Or is there an easier solution? Please let me know your thoughts on how to best optimize the pages
On-Page Optimization | | brightsocial0 -
Keyword Stuffing Question
Say your on a e-commerce category page "Shirts" every lower level category has "shirts" in it such as: T-shirt, long sleeve shirt, sweat shirt, v-neck shirt, and so on. Is this page going to be penalized in google for the keyword "shirts" just because it is in the title and on the page a thousand times because i'm targetting words like "long sleeve shirt? and if it is, will the "long sleeve shirt" keyword be negatively affected as well? Answer much appreciated,
On-Page Optimization | | Mike.Bean
Thanks in advance.0 -
Can the Lightboxes on My Site be Crawled?
I'm trying to optimize my site, but I have lightboxes and I don't know if they are visible to the search engines. If they aren't, could you suggest something that I could do? THANK YOU so much!!!!! My site is lymphexpo.com
On-Page Optimization | | bosleypalmer0 -
Improving the search function on my site...
Hi all, The search function on my site is pretty bad... it basically lists every single product for any search query. Has anyone got experience integrating a 3rd party tool such as Google Custom Search or Pardot and if so which would you recommend? Alternatively, any tips on improving/ creating rules for site search would be appreciated. https://www.boardwarehouse.co.uk/ Thanks, Alick
On-Page Optimization | | Alick3000 -
A question on cached pages
Hi all. I have been testing different ways of writing the text on the homepage of the website www document-management-solutions co.uk because we are ranking very low for the keyword Document Management Software. I've recently learnt about pages being 'cached' and wanted to just clarify something. The last time the page was cached was on 26th March. I have made changed to the homepage since then and wanted to see how it has affected our ranking. Will I not be able to know if it has had a positive/negative effect until the next time our page is cached? Thanks
On-Page Optimization | | janc0 -
Changing my site (dramatically)
I am about to do a complete site change. I am going to WordPress. I am ranked #2 on SERPS. Will I lose rank for changing everything on my site? I have 500 pages indexed but I am about to have 30k indexed. It is a real estate site that is switching from a "framed" solution, to a listing indexed solution. If I make good use of my keywords etc (on site optimization) will I be at risk of losing risk just for changing my site?
On-Page Optimization | | JML11790 -
Is there any benefit in on-site duplicate content?
I have about 50 internal pages on my site that I want to add a "Do it yourself tutorial" to in an effort to build the quality of the pages. Is this going to de-value the content if I put it on all 50 pages? It's difficult to write similar content 50 different ways.
On-Page Optimization | | BradBorst0 -
Webmaster tools Site speed?
Google webmaster tools site performance is reading out at 2.8 and still raising (Going further into the slow pale area) this was in the green fast area for a while until now. Is this something to be worried about?
On-Page Optimization | | BobAnderson0