Search directory - How to apply robots
-
Hi.
On the site I'm working on, we use a search directory to display our search results. It displays as follows -
With the dynamic search results appearing after the hash tag.
Because of the structure of the website, many of the lefthand nav defers back to this directory. I know that most websites "noindex, nofollow" the search results pages, but due to the ease of customers generating them, I'm afraid that if I do this, we'll miss out on the inevitable links customers will provide...and, even though it's just the main search directory, these links will still help my domain. The search is all java-generated so there's nothing for spiders to follow within this directory - save the standard category nav.
How should I handle this?
Thanks.
-
That was my first instinct as well - but I wanted to check if there was a better move? Thanks for your help!
-
Have you considered switching to simply "noindex, follow" for these pages? This would both keep them out of the search index and would allow for the benefit of external links to these pages.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why my website has problem in search result
hello everyone when i search "متیلن کلراید" on google sometimes my website is first page on google and sometime isn't on google. my website is https://nchemicalgroup.com/ help me please
Web Design | | sporting23231 -
Disallow: /sr/ and Disallow: /si/ - robots.txt
Hello Mozzers - I have come across the two directives above in a robots.txt file of a website - the web dev isn't sure what they meant although he implemented robots.txt - I think just legacy stuff that nobody has analysed for years - I vaguely recall sr means search request but can't remember. If any of you know what these directives do, then please let me know.
Web Design | | McTaggart0 -
Do SEOs really need to care about trend in increase of voice search?
Hi all, Just browsing through the articles like "SEO trends in 2108"....I can see everyone mention about increase in voice search and attributing same to the SEO at top. But there is very less information on how to optimise for voice search....May be, they don't know what SEOs have to do exactly with voice search. Basically the input method is going to change on search and else remains same. This must be leading to increase of long tail searches and more search snippets to give an instant voice answer. What else has to be optimised to favoured in voice search? Thanks
Web Design | | vtmoz2 -
Any second opinions as to why our organic search website traffic hasn't recovered from website rebrand (domain change, website redesign)?
I am hoping to see if anyone in the Moz community would be able to help troubleshoot or lend any advice on a major organic search traffic issue we've been experiencing over the last 8 months. In a nutshell, we decided our ~4.5-year-old business needed to undergo a rebrand in October 2015. After changing domains & redesigning our website (more below), our search-driven sessions have dropped 20% in 2016 v.s. 2015. We made quite a few on-site modifications (with some success) post-redesign but are still deep in a rut and not sure what more we can do to recover. I've listed my theories below as to why we're still suffering this hit. If anyone could weigh in on these and/or share any other troubleshooting ideas, I would greatly, greatly appreciate it (and owe you a lunch/beverage of your choice the next time I'm in your city!). ****Backlinks - despite our efforts to 301 all links, I sense we have lost many backlinks. According to Open Site Explorer, our old domain has 1,172 backlinks (some from some very authoritative pages domains), 1,068 of which are passing link equity. In contrast, our new domain has 367 backlinks, 321 are passing link equity, and very few overlap with our old domain. Domain Age - we may have lost much of our reputation with Google as our new domain is much younger than our old domain (1-year-old v.s. 5.5 years old). Domain Name - although I thought to have common keywords in one's domain was a myth, I am now questioning that belief. Our old domain contained a popular, topical keyword and our new domain is derived from a term that is topical, but very uncommon. New URLs - our developer has insisted all links were moved to the new domain, but I have a hunch they were not. When conducting a "site search" (i.e. "site:websitename.com"), the new domain returns 7,740 results. Prior to our switch, a site search with the old domain yielded 30,000+ results. 404s - we found and fixed 100-200 404'd links after the domain switch. We still see a few pop-up today and I'm wondering if this is a red flag in Google's eyes. For a little more background too, here are the nitty gritty details with a rough timeline: Pre-October 12, 2015 - registered new domain and designed the new website on Wordpress, while researching a range of articles and resources for a successful site migration (e.g. this and this Moz guide). October 12, 2015 - flipped the switch on the website design, domain, minor content reorganization, and social handles. We announced the change to our audience via an article, newsletter, and social; informed Google Webmaster Tools (GWT) of the new address, 301'd all links from the old to the new domain, and submitted new sitemap in GWT. October 12 - 16, 2015 - traffic is normal, everything seems to be okay. October 17, 2015 - search traffic drops by 54% v.s. the same day of week pre-rebrand. October 26, 2015 - search traffic rises, so now only down by 30% v.s. the same day of week pre-rebrand. November/December 2015 - re-added numerous elements from the old website such as category, tag, and page pagination and a few sidebar modules that linked to other important pages and tags. Search traffic rises slightly in November (down 27% year-on-year), dips again in December (down 31% year-on-year). January 2016 - today (June 17, 2016) - we published more content on a daily basis and search traffic fluctuates around the 20% versus the same period in 2015. January 2016 - down 23% year-on-year February 2016 - down 17% year-on-year March 2016 - down 20% year-on-year April 2016 - down 21% year-on-year May 2016 - down 21% year-on-year June 2016 (until the 17th) - down 23% year-on-year Thank you all in advance for your time and help, please let me know if you have any questions!
Web Design | | nick490 -
Robots.txt being blocked
I think there is an issue with this website I'm working on here is the URL: http://brownieairservice.com/ In Google Webmaster tools I am seeing this in the Robots.txt tester: User-agent: *
Web Design | | SOM24
Crawl-delay: 1
Disallow: /wp-content/plugins/
Disallow: /wp-admin/ Also when I look at "blocked resources" in the webmaster tools this is showing to be blocked: http://brownieairservice.com/wp-content/plugins/contact-form-7/includes/js/jquery.form.min.js?ver=3.51.0-2014.06.20It looks like the form plug in is giving the issues but I don't understand this. There are no site errors or URL errors so I don't understand what this crawl delay means or how to fix it. Any input would be greatly appreciated. Thank you0 -
Hi, I have a doubt. If we want to hide unwanted text in a web page its possible with "" tag. And my question "does a search engine crawl those text? help me.
I want to hide a lot of text behind my site page. I know its possible with that tag. But in what way a search engine looks at those text? Hidden or they are crawled and indexed.
Web Design | | FhyzicsBCPL0 -
Map Search Tools to integrate on Accommodation Website
Hello all, Can any recommend a Map Search tool that i can integrate into my accommodation website. Ideally I want to be able to pin my clients on the map search with links back to their listing page on my website and provide an alternative search facility for clients looking for accommodation. Am covering the South Africa region specifically. Am assuming that i could go down the Google Maps route but would really like to know what alternatives there are on offer. Also what SEO considerations do i need to think about when adding this to my website. Thanks in advance for any help.
Web Design | | SamanthaRiggien0
