Robots.txt Question for E-Commerce Sites
-
Hi All,
I have a couple of e-commerce clients and have a question about URLs.
When you perform a search on website all URLs contain a question mark, for example: /filter.aspx?search=blackout
I'm not sure that I want these indexed.
Could I be causing any harm/danger if I add this to the robots.txt file? /*?
Any suggestions welcome!
Gavin
-
You're right on target, it's not a good idea to index search results. Google doesn't want to crawl or index other search results in its own search results. There are some exceptions for gigantico sites like Yelp or TripAdvisor when showing their search results pages are actually the best option, but if you're not at that level and especially if you're an ecommerce site, it's not recommended.
You wouldn't be harming anything by excluding search from your robots.txt file. In fact, many top sites exclude search results to preserve crawl capacity and for indexation reasons.
You'll also want to look at parameter handling in Search Console, this article from Google will get you started.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Site name in page title - leave it or remove it?
Hi all, Recently came across some authority blog (quicksprout to be precise) which stated that apart from main page, contact page, about us and some other generic pages, site name should be removed as it might produce duplicate content. example "How to blog | Example Site name" This mostly is the issue with tags and categories pages as it shows on Moz issues. Is that really a problem and site name should be taken off them? Thank you.
On-Page Optimization | | Optimal_Strategies1 -
Question For SEO Companies From An On-line Retailer
When employing an SEO company what should I be looking for? When I go to a companies website I look for how many keywords they have ranked on Semrush. Most of the companies that I have looked at do not rank very many. Saying most I am talking about 10-12 companies. There was one company that had a lot more than anyone else but their reviews were awful. Then I look at their clients websites to see how many keywords they have ranked and this is where I am baffled. Quite a lot of them especially retail sites with lots of products have very few keywords ranked and my belief is that the more related keywords that you have ranked the better you will do. I also look at how the thier organic traffic is doing over a number of months. what surprises me most is that companies are advertising clients websites where it looks like traffic is climbing and then all of a sudden there is a big drop. Am I missing something in order to find a company that will do what we are paying them for? Thank you in advance.
On-Page Optimization | | Palmbourne0 -
Acquired Old, Bad Content Site That Ranks Great. Redirect to Content on My Site?
Hello. my company acquired another website. This website is very old, the content within is decent at best, but still manages to rank very well for valuable phrases. Currently, we're leaving the entire site active on its own for its brand, but i'd like to at least redirect some of the content back to our main website. I can't justify spending the time to create improved content on that site and not our main site though. What would be the best practice here? 1. Cross-domain canonical - and build the new content on our main website? 2. 301 Redirect Old Article to New Location containing better article 3. Leave the content where it is - you won't be able to transfer the ranking across domain. Thanks for your input.
On-Page Optimization | | Blenny0 -
Is there a way to tell Google a site has duplicated content?
Hello, We are joining 4 of our sites, into 1 big portal, and the content from each site gonna be inside this portal and sold as a package. We don't wanna kill these sites we are joining at this moment, we just wanna import their content into the new site and in a few months we will be killing them. Is there a way to tell Google to not consider the content on these small sites, so the new site don't get penalised? Thanks,
On-Page Optimization | | darkmediagroup0 -
Question Regarding RSS Feeds
A site I have been asked to take a look at has an RSS feed on there home page. The site offers Austrian mortgages. But the RSS feed is in the side bar and it is drawing information for Austrian News. I personally would of said that if it was to draw anything then it should be more accurately related to the site, or am I wrong? I also have seen that he has installed RSS Bomber, now it just submits his blog posts to a lot of RSS aggregators each time he makes a post. Is this a bad thing as I am not sure. I am thinking it cant be good as it is effectively automated but I would like others opinions on this. Thanks Paul
On-Page Optimization | | propertyhunter0 -
Which pages on my site should I back link to
The majority of the back links I have been creating link directly to our home page and to the store page. Is this the best approach or should I be trying to spread the links throughout our site to include product categories and subcategories etc?
On-Page Optimization | | Hardley0 -
Site Architecture: How do I best Optimize for Similar Keywords?
Hello Moz Community! I'm really struggling trying to decide on an improved site architecture. I run an online proofreading & editing website. This leaves us targeting many different niche keywords. For example: blog editing/proofreading, essay editing/proofreading, book editing/proofreading, resume... you get the point. I feel like editing & proofreading are similar enough to target on the same page(s). However, the issue is that I'm also having to deal with what I'm calling derivative keywords. For example, when I try to optimize for 'essay editing/proofreading', I also have to think about: paper editing, paper editor, paper correction, edit my paper, etc. I would have no problem optimizing the page for 'essay editing' in the title, H1, etc. and then targeting these words as secondary keywords within the body text, etc., however, I keep thinking 'a large slice of a small pie is better than a small slice of a big one.' You see, the keyword 'essay correction' has only about one-third the monthly searches as 'essay editing', but it is 50% less competitive. The same is loosely true for the rest of the 'derivative' keywords. I'd have no problem building specific pages for these derivative keyword groups, however, I'm very concerned how this would effect my site from a user experience perspective. I don't want to have a master "services" page with links to book editing, resume editing, essay editing, etc. and then also show paper editing, essay correction, etc. To me, this would be confusing... "What's the difference between essay editing and paper editing?". Any guidance is much appreciated. This has got my head spinning! Thanks!
On-Page Optimization | | TBiz0 -
Robots.txt: excluding URL
Hi, spiders crawl some dynamic urls in my website (example: http://www.keihome.it/elettrodomestici/cappe/cappa-vision-con-tv-falmec/714/ + http://www.keihome.it/elettrodomestici/cappe/cappa-vision-con-tv-falmec/714/open=true) as different pages, resulting duplicate content of course. What is syntax for disallow these kind of urls in robots.txt? Thanks so much
On-Page Optimization | | anakyn0