Block all search results (dynamic) in robots.txt?
-
I know that google does not want to index "search result" pages for a lot of reasons (dup content, dynamic urls, blah blah). I recently optimized the entire IA of my sites to have search friendly urls, whcih includes search result pages. So, my search result pages changed from:
- /search?12345&productblue=true&id789
to
- /product/search/blue_widgets/womens/large
As a result, google started indexing these pages thinking they were static (no opposition from me :)), but i started getting WMT messages saying they are finding a "high number of urls being indexed" on these sites. Should I just block them altogether, or let it work itself out?
-
You can block the urls which has term "/product/search/" in them. It can be easily done by adding the following to the robots.txt
User-agent: * Disallow: /product/search/ Hope this helps...
-
As you said: The increasing number of pages indexed will dilute the link juice of the entire site.
Can you give more example? Or just a tip where to search for this kind of information?
Thank you.
-
I would agree with BK Search. You want to minimize what Google has to crawl (I know this sounds backwards) so that Google focuses on the pages that you want to rank.
Long term, why would you waste GoogleBot's time on pages that don't matter as much? What if you had an update on a more important page and GoogleBot is too busy indexing this infinite loop of pages.
At this point, I would use the noindex meta tag vs robots.txt so that google will crawl and remove all the urls from the index. Then you can drop it in later into robots.txt so it will stop crawling. Otherwise you may end up with a lot of junk in the index.
-
I might be a little different than some of these answers but I would recommend that you exclude them from getting indexed.
The reasons I would do that are that:
You know it is largely duplicate content and goes down to the same pages as your categories.
Google has stated that they would prefer to not have it indexed.
The increasing number of pages indexed will dilute the link juice of the entire site.
There is also the possibility that people using the url bar of their browser will start to increase the number of pages indexed by a large manner.
A competitor could create thousands of links to these pages and create a huge footprint that is search pages.
And finally, I like having product pages ranking highly if at all possible.
I would do this with both the robots.txt file and the GWMT exclusion on /product/search/ directory
Good Luck!
-
Hi! We're going through some of the older unanswered questions and seeing if people still have questions or if they've gone ahead and implemented something and have any lessons to share with us. Can you give an update, or mark your question as answered?
Thanks!
-
As a follow-up or further info: Its been about 5 months since the change. I do get some traffic from these indexed pages (not a ton, but enough that i would like to not block if there is no negative impact). The SE interaction seems to be confusion- they index the content, but also recognize that something may not be right. So I am wondering if anyone else has done something similar or is trying this.
Admitidly this is what i wanted the new url structure to do- as an experiment. Just looking for anyone else who has/is doing similar
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Robots.txt Help
I need help to create robots.txt file. Please let me know what to add in the file. any real example or working example.?
Intermediate & Advanced SEO | | Michael.Leonard0 -
Sitelinks in non-brand based organic search results
Hi all, I have a question for everyone. Sitelinks have been around for a while now & I've always seen them when the search is for a brand's name. However, today, when looking at the rankings for one of the campaigns we manage, we noticed there were sitelinks in the number #1 & #2 positions in Google (Australia) for the search term "Dance Costumes". Whilst both the companies have Dance Costumes in their title, so do all the other results & so I don't see why it warrants the sites to be relevant via their brand name.
Intermediate & Advanced SEO | | KBB_Digital
Note: The results are organic results, not paid results (where you can add sitelinks). Firstly, has anyone seen this before (screenshot attached)?
And secondly, is there markup/schema that allows you to do this (none that I know of)? danceCostumes-sitelinks.png0 -
Robots.txt vs noindex
I recently started working on a site that has thousands of member pages that are currently robots.txt'd out. Most pages of the site have 1 to 6 links to these member pages, accumulating into what I regard as something of link juice cul-d-sac. The pages themselves have little to no unique content or other relevant search play and for other reasons still want them kept out of search. Wouldn't it be better to "noindex, follow" these pages and remove the robots.txt block from this url type? At least that way Google could crawl these pages and pass the link juice on to still other pages vs flushing it into a black hole. BTW, the site is currently dealing with a hit from Panda 4.0 last month. Thanks! Best... Darcy
Intermediate & Advanced SEO | | 945010 -
Something unintelligible in google search results.
In the last month i see something that i can't explain, from nowhere the first search result for the keyword Name Necklace is an Etsy shop, (I attach an image). I try to find out why google gives this online shop the first position and i have no idea why is that. Please, any suggestions? mnbAIrY.png
Intermediate & Advanced SEO | | Tiedemann_Anselm0 -
Do these results indicate a problem with my seo?
I've entered my the following search query into Google.co.uk related:mywebsite.co.uk However the resulting website that are brought back are on the whole nothing like our website, nor do they offer similar services to us. If I run this same query on my competitors websites they all bring back similar websites to each other. I read somewhere that gaining links from the websites that Google believes are similar/related to our own website is beneficial. But looking at our results it would seem that Google can't place what our site is about and which sites are similar. So I'm guessing this is a more pressing matter than link building right now!? Other info about our website: We rank fairly well for a lot of our target keywords.
Intermediate & Advanced SEO | | adamlcasey
Domain age = 11 years
PA =38
mR= 4.77
mT= 5.74
DA:= 31
DmR= 3.78
DmT= 3.84
PageRank = 3 Example of how random the results are the 1st website that comes back in our related websites search is for Doctors GP Practice. Our website sells GPS Telematics Solutions. Can anyone shed any light on this or just to confirm how much of a problem this is?0 -
Anyone managed to change 'At a glance:' in local search results
On Google's local search results, i.e when the 'Google places' data is displayed along with the map on the right hand side of the search results, there is also an element 'At a glance:'
Intermediate & Advanced SEO | | DeanAndrews
The data that if being displayed is from some years ago and the client would if possible like it to reflect there current services, which they have been providing for some five years. According to Google support here - http://support.google.com/maps/bin/answer.py?hl=en&answer=1344353 this cannot be changed, they say 'Can I edit a listing’s descriptive terms or suggest a new one?
No; the terms are not reviewed, curated, or edited. They come from an algorithm, and we do not help that algorithm figure it out. ' My question is has anyone successfully influenced this data and if so how.0 -
Can I compete with these results? (Brand in Serp)
Hey, One quick question. Lets say im fighting for keyword "british airways" and i want to appear straight after first result in number 2 position. Is it possible to compete with stroked results. (See image attached) Thanks Stxct.png
Intermediate & Advanced SEO | | Marteen0 -
Block an entire subdomain with robots.txt?
Is it possible to block an entire subdomain with robots.txt? I write for a blog that has their root domain as well as a subdomain pointing to the exact same IP. Getting rid of the option is not an option so I'd like to explore other options to avoid duplicate content. Any ideas?
Intermediate & Advanced SEO | | kylesuss12