Block all search results (dynamic) in robots.txt?
-
I know that google does not want to index "search result" pages for a lot of reasons (dup content, dynamic urls, blah blah). I recently optimized the entire IA of my sites to have search friendly urls, whcih includes search result pages. So, my search result pages changed from:
- /search?12345&productblue=true&id789
to
- /product/search/blue_widgets/womens/large
As a result, google started indexing these pages thinking they were static (no opposition from me :)), but i started getting WMT messages saying they are finding a "high number of urls being indexed" on these sites. Should I just block them altogether, or let it work itself out?
-
You can block the urls which has term "/product/search/" in them. It can be easily done by adding the following to the robots.txt
User-agent: * Disallow: /product/search/ Hope this helps...
-
As you said: The increasing number of pages indexed will dilute the link juice of the entire site.
Can you give more example? Or just a tip where to search for this kind of information?
Thank you.
-
I would agree with BK Search. You want to minimize what Google has to crawl (I know this sounds backwards) so that Google focuses on the pages that you want to rank.
Long term, why would you waste GoogleBot's time on pages that don't matter as much? What if you had an update on a more important page and GoogleBot is too busy indexing this infinite loop of pages.
At this point, I would use the noindex meta tag vs robots.txt so that google will crawl and remove all the urls from the index. Then you can drop it in later into robots.txt so it will stop crawling. Otherwise you may end up with a lot of junk in the index.
-
I might be a little different than some of these answers but I would recommend that you exclude them from getting indexed.
The reasons I would do that are that:
You know it is largely duplicate content and goes down to the same pages as your categories.
Google has stated that they would prefer to not have it indexed.
The increasing number of pages indexed will dilute the link juice of the entire site.
There is also the possibility that people using the url bar of their browser will start to increase the number of pages indexed by a large manner.
A competitor could create thousands of links to these pages and create a huge footprint that is search pages.
And finally, I like having product pages ranking highly if at all possible.
I would do this with both the robots.txt file and the GWMT exclusion on /product/search/ directory
Good Luck!
-
Hi! We're going through some of the older unanswered questions and seeing if people still have questions or if they've gone ahead and implemented something and have any lessons to share with us. Can you give an update, or mark your question as answered?
Thanks!
-
As a follow-up or further info: Its been about 5 months since the change. I do get some traffic from these indexed pages (not a ton, but enough that i would like to not block if there is no negative impact). The SE interaction seems to be confusion- they index the content, but also recognize that something may not be right. So I am wondering if anyone else has done something similar or is trying this.
Admitidly this is what i wanted the new url structure to do- as an experiment. Just looking for anyone else who has/is doing similar
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Related searches volume
Hello, The related searches we see at the bottom of google search results. Is this what people also search for ? If it is why does the keyword so no volume for those related searches ? Thank you,
Intermediate & Advanced SEO | | seoanalytics0 -
Desktop vs. Mobile Results
When googling on www.google.ca for "wedding invitations" and in my own geo location market of Toronto, my site - www.stephita.com, will show up differently on SERP on desktop (Chrome & IE) vs. mobile (iPad, iPhone, android, etc.). On desktop SERP, I will show up 6/7 position... (which is relatively a new position, the past 3 weeks - I was previously on page 2) (After a bunch of SEO fixes, I've managed to propel my site back to page 1!) On mobile SERP, I only show up on 1/2 position on PAGE 2 😞 As I mentioned above, I did a bunch of SEO fixes that I think were related to Panda/Penguin algos. So I'm wondering why my MOBILE SERP has NOT improved along the way? What should I be looking at to fix this 5-6 position differential? Thanks all!
Intermediate & Advanced SEO | | TysonWong0 -
Robots.txt gone wild
Hi guys, a site we manage, http://hhhhappy.com received an alert through web master tools yesterday that it can't be crawled. No changes were made to the site. Don't know a huge amount about the robots.txt configuration expect that using Yoast by default it sets it not to crawl wp admin folder and nothing else. I checked this against all other sites and the settings are the same. And yet 12 hours later after the issue Happy is still not being crawled and meta data is not showing in search results. Any ideas what may have triggered this?
Intermediate & Advanced SEO | | wearehappymedia0 -
Can URLs blocked with robots.txt hurt your site?
We have about 20 testing environments blocked by robots.txt, and these environments contain duplicates of our indexed content. These environments are all blocked by robots.txt, and appearing in google's index as blocked by robots.txt--can they still count against us or hurt us? I know the best practice to permanently remove these would be to use the noindex tag, but I'm wondering if we leave them they way they are if they can still hurt us.
Intermediate & Advanced SEO | | nicole.healthline0 -
Blocked from google
Hi, i used to get a lot of trafic from google but sudantly there was a problem with the website and it seams to be blocked. We are also in the middle of changing the root domain because we are making a new webpage, i have looked at the webmaster tools and corrected al the errors but the page is still not visible in google. I have also orderd a new crawl. Anyone have any trics? do i loose a lot when i move the domainname, or is this a good thing in this mater? The old one is smakenavitalia.no The new one is Marthecarrara.no Best regards Svein Økland
Intermediate & Advanced SEO | | sveinokl0 -
Losing Rank As A Result Of Domain Change
I have a client who is wishing to switch from an established, but unattractive domain, to a domain he just purchased that is more attractive. For example purposes, his existing site is "His-Company-Website.com" and the site he just purchased and wants to transfer to is "HisCompanyWebsite.com." The only difference is the old site has hyphens in between the words and the new one does not. He is not making this choice from an SEO perspective, but more of a "I don't want to keep saying all those hyphens when telling people about my website." But he said he doesn't want to lose his search engine rankings as a result. So he knows this won't necessarily increase his ranks, but doesn't want them to drop as a result. When speaking with him, I thought we could simply toss in a 301 redirect at the root level and pipe them over to the other site, but he wanted some actual proof. I went back to look at what I thought would be a similar case that I did earlier in the year (transferring from a .net to a .com) and noticed that we did see some rather substantial drops in at least traffic, so I am not so sure about this plan any longer. So my questions for my far more insightful colleagues... What would be your suggestion on this problem? Transition to the more user friendly domain or stick with the unfriendly domain? If he does elect to transition to the new domain, what all can I do to preserve his search engine rankings? Should a rankings and/or traffic drop be predicting when completing this? Thank you all in advance. Any other tidbits anyone has to offer would be great. Looking forward to your replies.
Intermediate & Advanced SEO | | ClayPotCreative0 -
Universal Search vs Local Organic
Hi, My web site has high rankings in universal SERP's. However, in my city organic search the competitors’ web sites that even don’t show up in universal Serp’s have higher rankings than mine. Not sure what I’m doing wrong. Thanks for any insight.
Intermediate & Advanced SEO | | Zlhe0 -
Could you use a robots.txt file to disalow a duplicate content page from being crawled?
A website has duplicate content pages to make it easier for users to find the information from a couple spots in the site navigation. Site owner would like to keep it this way without hurting SEO. I've thought of using the robots.txt file to disallow search engines from crawling one of the pages. Would you think this is a workable/acceptable solution?
Intermediate & Advanced SEO | | gregelwell0