Robots.txt Question for E-Commerce Sites
-
Hi All,
I have a couple of e-commerce clients and have a question about URLs.
When you perform a search on website all URLs contain a question mark, for example: /filter.aspx?search=blackout
I'm not sure that I want these indexed.
Could I be causing any harm/danger if I add this to the robots.txt file? /*?
Any suggestions welcome!
Gavin
-
You're right on target, it's not a good idea to index search results. Google doesn't want to crawl or index other search results in its own search results. There are some exceptions for gigantico sites like Yelp or TripAdvisor when showing their search results pages are actually the best option, but if you're not at that level and especially if you're an ecommerce site, it's not recommended.
You wouldn't be harming anything by excluding search from your robots.txt file. In fact, many top sites exclude search results to preserve crawl capacity and for indexation reasons.
You'll also want to look at parameter handling in Search Console, this article from Google will get you started.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Question about my website ranking
Hi everyone I have sports website. My website DA, DR is good. I have created good quality backlinks. But still my website ranking is not improving. Can you guide me how to improve my website ranking. Thanks in advance!
On-Page Optimization | | fktos.tam.906n0 -
Redirecting pages (old site to new site)
I have a question- there is one location, one set of pages for both the old and new site on the same host environment so when I did the redirect it get into a loop trying to redirect from itself to itself Not sure how its gonna affect SEO. Will pages get hit for duplicate content?
On-Page Optimization | | Yanez0 -
SEO for New Magento Site??!?
Hi All, We have had a new site delivered by our developers (hurrah...the old one was to terrible for words) but we seem to be having a lot of issues with the new Magento platform. What they have done is used the community version and tried to customize it. I came on to this project about a month into the build and although Magento does seem to do a lot things well there do seem to be some problems. From an SEO prespective we have seen some increase on some search terms and a drop of in others. I would be interested in hearing from other Magento users about their experiences with this platform and any ideas on how to crank up the activity. Our site is at www.nationwidepharmacies.co.uk . There are few odd bits including the side navigation which seems to be very clunky and not overly customizable in this version. Any useful criticism would also be well received. Look forward to hearing Nic
On-Page Optimization | | nicc19760 -
Blocking Pages E-Commerce Site
Hello, I am working on a site with 1,000's of product pages, some of which do not have inventory in them. Should be blocking these pages in order to reduce bouce rate? How could i manage so many pages efficiently? It would takes weeks to comb through pages to determine which have inventory and which do not. They are also time sensitive as they are live events so dates are always changing. Thanks!
On-Page Optimization | | TP_Marketing0 -
Site Downtime - Will pages be reindexed?
I recently had site downtime of about 10 days for a site I'm working on which is about 6 months old. There were naturally many page not found errors in webmaster tools as google had tried to crawl the site during the downtime but I've noticed that google has now dropped some of the pages. Are they likely to be reindexed? Google has crawled some of my pages today but as yet the ones that have been dropped from the index haven't reappeared. (the site has been live again for a week). Should they reappear when the pages are next crawled?
On-Page Optimization | | SamCUK0 -
Product Title Formating Question
The majority of the products I sell require lengthy product titles, I have 600+ items that have titles over 70+ characters. In the interest of reducing the character count (Product Feeds) I'm trying to eliminate unnecessary "stop words" in the titles, such as "and", "with" etc.. I've listed an example of a current product listing below, followed by two different formats to reduce the character count. Which one if any is better for SEO or should I leave my titles alone and stick with my original format? Current Listing Example: DeWalt Dual Comfort Safety Glasses with Black Frame and Clear Anti-Fog Lens Suggested Listing Example: DeWalt Dual Comfort Safety Glasses-Black Frame-Clear Anti-Fog Lens DeWalt Dual Comfort Safety Glasses/Black Frame/Clear Anti-Fog Lens
On-Page Optimization | | MEldridge0 -
Very basic hands-on type of question about SEO
Hello, I am a complete newbie to the world of SEO and I have read few thing available on the net about things to begin with (so google seo guide and then seomoz beginers tutorial) and I just wanted to ask if I understand the process correctly: So I have my website lest call it abcd.com and there I might have a subpage about certain type of robots or specific parts that these robots are built from. I do my keyword search using adwords keyword search tool. Where I get 10 phrases (phrase1, phrase2, phrase3 so on..) that could be used by users to search information about certain type of robots I wrote on my website. Let's asume all 10 of those phrases have low competition so they can all be used. And they are from the long tail ofcourse so let's say I can get 10,000 searches from them per month. Some of them would have 200 searches, some 1500, some 5000 per month in that google adwords report. After reading those basic tutorials I understand it this way: I put one or two best phrases of those 10 in my <title></p> <p>2) I describe the website as accurately as possible to those keywords in <meta description> i.e. Those specific robots are built from those specific parts - we know all about them.</p> <p>3) I put all of them in <meta keywords> phrase1,2,3, etc.</p> <p>4) I use one or two phrases in <h1> on the page with article</p> <p>5) then I use those phrases in the text that's on that particular page of my website - text is about certain type of robots and parts etc.</p> <p>6) I put photos on that page with alt descriptions that may contain some of these phrases</p> <p>7) to be honest I don't know how am I suppose to build links from that page about certain type of robots to anywhere else on my website - but I undestand that's the part of SEO as well</p> <p>And that's pretty much how I understand the basics of SEO. I read about it and I just don't know if that's what I am suppose to do. Silly me!</p> <p> </p></title>
On-Page Optimization | | lolskizz1 -
Site URL's
We are redeveloping our website, and have the option to amend URLs (with 301 redirects from old URL to new), so my question is: Would 'golfsite.com/golf-clubs' achieve superior rankings than 'golfsite.com/clubs' for the search term 'golf clubs' if all other factors were the same? Should the URL reflect the intended search term wherever possible?
On-Page Optimization | | swgolf1230