Robots.txt Question for E-Commerce Sites
-
Hi All,
I have a couple of e-commerce clients and have a question about URLs.
When you perform a search on website all URLs contain a question mark, for example: /filter.aspx?search=blackout
I'm not sure that I want these indexed.
Could I be causing any harm/danger if I add this to the robots.txt file? /*?
Any suggestions welcome!
Gavin
-
You're right on target, it's not a good idea to index search results. Google doesn't want to crawl or index other search results in its own search results. There are some exceptions for gigantico sites like Yelp or TripAdvisor when showing their search results pages are actually the best option, but if you're not at that level and especially if you're an ecommerce site, it's not recommended.
You wouldn't be harming anything by excluding search from your robots.txt file. In fact, many top sites exclude search results to preserve crawl capacity and for indexation reasons.
You'll also want to look at parameter handling in Search Console, this article from Google will get you started.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
When does updating site content affect SERPS?
Can any of you shine a light as to when updating content on a website had any effect on its 'general' placement in the search engines? I appreciate that for an online newspaper, it must be important, but has anyone noticed from theirs or a client's site that by not uploading i.e. product descriptions or articles in general, that their site has taken a hit?
On-Page Optimization | | SDavis110 -
Online classified ads site - duplicate content?
Hello, I was reading hobo s post on duplicate content. Our web is in the classified advertisement industry and our site is built up like this Homepage (last 200 ads) category 1(has the name we want to rank our homepage and around 350 ads) category 2 (around 100 ads) category 3 (around 60 ads) Now our homepage has 200 ads that also appear mostly in category 1 but also in others. We are ranking our homepage as 11 th now on Google. I'm worried a bit that the 200 ads on the homepage are not unique, because they will appear in one other category. Is this OK? Is this duplication? Should we do something? Issue is that we at first started ranking our homepage where all ads were, now there are too many so we show 200 latest on homepage and then they are split into category pages.
On-Page Optimization | | advertisingcloud0 -
Translated the site but traffic is not coming
Hi, We've build a lawyer directory website (www.iranianlawyers.com) which already has good Google rankings for related terms (Ex. Iranian lawyers, Iranian lawyers california, etc). About 1.5 months ago we translated the site to Farsi and published it online: www.iranianlawyers.com/farsi However we have yet to see any new traffic generated from those pages. The website has a decent back-link profile and there are no almost no competitors in our space with translated pages. Would someone please take a look at our translated pages and let me know if there are any major on-site issues that you see that we need to address? I've checked for noindex or nofollow tags but they dont seem to be an issue. Not sure if I'm missing something here. Thank you very much
On-Page Optimization | | Heydarian0 -
On-Page Optimization Question
My company sells Blue widgets and we are located in Denver, CO. Keyword research indicates that the the highest volume phrase is "blue widgets for sale in denver co". Should my meta title tag be: Blue Widgets for sale in Denver CO , and my h1 tag be the same? or should they be semantic phrases? Thanks in advance!
On-Page Optimization | | FicklingCompany0 -
Does my site just need a tweak, or should it be rebuilt from the ground up?
My wedding photography website has grown organically over the last 4 years or so.
On-Page Optimization | | chuckstar_za
Each year it's been fiddled with, changed, tweaked, prodded - in short it's been mucked about with a lot.
To the point now where I think it's become a hodge potch of mucky code (theme installed on theme in stalled on theme and numerous plugins) and no real 'structure' to it. At the moment it hovers around #12 for my main area of focus - people looking for a wedding photographer in Essex, England. So 'wedding photographer Essex'. I try to write blog posts regarding venues in Essex where I've photographed weddings. Despite trying to follow best practice, and with weak/low competition, these posts very rarely crack top 10. Last year when I was top 2 for wedding photographer Essex I was victim of a spam link attack. At the time I asked for advice and was told not to worry as Google didn't count these links. Now I'm feeling that along with the bloated website, lack of proper site architecture and other little issues, these links are contributing to restricting my pages from ranking in top 10 spots. Backlinking, while always helpful, doesn't seem to play a major role when trying to rank for venue related searches according to backlink reports on competitors. I'm thinking that it might be an idea to rebuild the site and content from the ground up. This time with a proper site structure in place and conforming to best practice. Obviously this is probably the most extreme course of action. Or does my site just need a little TLC to get it ranking better?
The site in question is:
www.alexkilbeephotography.co.uk Thanks1 -
Moving from Local to National Audience - Ecommerce Site
I have an e-commerce site that was optimized for local results. We’ve been receiving traffic from all over the nation so recently we've decided to optimize the site for a more national audience by removing the local business indicators. I have below a list of things that I’ve done, I’m contemplating doing and other various questions. One note I want to make is that we offer installation services – which we would still like to rank locally for. I have done the following: Remove [city] from all of the title tags (except for the installation pages) Removed NAP (name, address, ph#) from site wide footer – I’m not sure if I should have done this. Inserted NAP at the bottom of the body content of the installation pages Questions Right now my installation pages are located in a sub-folder (ex: www.mydomain.com/installation-services). Should I create a sub-domain for installations (ex: city.mydomain.com/installation-services)? My thoughts are that it would make that sub-domain all about our installation services while keeping our root domain focused on our products and not dilute it with installations. However, this sub-domain will have less juicy juice. What should I do with our G+ business page? In the back end of the business G+, there are boxes I can check for “my business has service areas where I visit my customers at their location” and “I serve customers at my business address.” If checked, will these make my site less likely to rank Nationally? We DO have a storefront and we DO visit people to do installations, so I would like these to be checked but not if it hurts my national rankings. There are local “7 pack” results for my main keywords and their installations. So, I would want my root domain to show up locally for “widgets in [city]” and my sub-domain to show up for “widget installation in [city]". Is it against Google’s TOS to create another G+ business page for the sub-domain? Is this even possible? Do I even need to do this or will Google know that I have a sub-domain all about installation and have my site show up in the local results? Is there anything in GWT that I need to make sure to have done? I put the geographic target to United States. Is there anything else? Of course, there is always the option of creating a new domain and optimize it for the installations. Any other suggestions would be greatly appreciated. Thank you.
On-Page Optimization | | SWWebTeam0 -
Silly question about noindex and canonical
Hi, This is probably going to sound a bit stupid, but I nevertheless want to check. We have a site that's going to have identical pages (really not my choice) for a sales reason. The two examples would be example.com/profile-name and example.com/location/profile-name Users using the onsite navigation will always end up in the latter example naturally as they have to select a location before viewing content (plus having the location in the url is nice as there are multiple profiles across different locations that have the same name). However, it's easier to sell our services when we can offer just example.com/profile-name to users for their own marketing reasons. I'd like to make the example.com/profile-name noindex follow, and have just the example.com/location/profile-name indexed, but not sure if it would be better to implement canonical tags instead? Can anyone see any potential pitfalls of using either method or does it not really make a difference (which is what I suspect, but I'd rather look stupid than get this wrong)? Thanks!
On-Page Optimization | | LeahHutcheon0 -
Good Site Navigation verses Success
I have been experimenting with the number of links on our pages verses the number of hits we get. Success seems to be tied to having hundreds of links on a page verses ease of navigation. We have a research company that sells research on educational topics. Last November I decided to divide our category of literature research topics into 10 different subtopics and redistribute the links to the subtopic pages. The main literature research page had over 800 links on it. It was one of our top performing pages. I was hoping that by spreading the links out in logical categories i could distribute the wealth and have better navigation for the user. Now after 6 months the traffic to that page has dropped 800% and the sub-pages have only gained a very minimal percentage. Overall, the hits in the literature genre have dropped from 560 per month to around 80. Ouch! I thought Google would love this strategy, as it reduced page load time, links on a page and made the navigation logical and easier to see all available options. Not the case. Question is: Should I keep the subpages but go back and put all the links back on the main literature page, putting it back up to 800 links? Should I get rid of the subpages, because the links will all be on the main literature page if I move them? Any advice is appreciated! Karen
On-Page Optimization | | eworld0