Search Results Pages Blocked in Robots.txt?
-
Hi
I am reviewing our robots.txt file. I wondered if search results pages should be blocked from crawling?
We currently have this in the file
/searchterm* Is it a good thing for SEO?
-
HI Dirk,
Sorry I missed this one, thanks for your input
-
It's probably a good thing - I would keep them blocked.
Check https://www.mattcutts.com/blog/search-results-in-search-results/ - quote "Use robots.txt to prevent crawling of search results pages or other auto-generated pages that don’t add much value for users coming from search engines."
Dirk
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What happens to crawled URLs subsequently blocked by robots.txt?
We have a very large store with 278,146 individual product pages. Since these are all various sizes and packaging quantities of less than 200 product categories my feeling is that Google would be better off making sure our category pages are indexed. I would like to block all product pages via robots.txt until we are sure all category pages are indexed, then unblock them. Our product pages rarely change, no ratings or product reviews so there is little reason for a search engine to revisit a product page. The sales team is afraid blocking a previously indexed product page will result in in it being removed from the Google index and would prefer to submit the categories by hand, 10 per day via requested crawling. Which is the better practice?
Intermediate & Advanced SEO | | AspenFasteners1 -
Pages blocked by robots
**yazılım sürecinde yapılan bir yanlışlıktı.** Sorunu hızlı bir şekilde nasıl çözebilirim? bana yardım et. ```[XTRjH](https://imgur.com/a/XTRjH)
Intermediate & Advanced SEO | | mihoreis0 -
Fixing A Page Google Omits In Search
Hi, I have two pages ranking for the same keyword phrase. Unfortunately, the wrong page is ranking higher, and the other page, only ranks when you include the omitted results. When you have a page that only shows when its omitted, is that because the content is too similar in google's eyes? Could there be any other possible reason? The content really shouldn't be flagged as duplicate, but if this is the only reason, I can change it around some more. I'm just trying to figure out the root cause before I start messing with anything. Here are the two links, if that's necessary. http://www.kempruge.com/personal-injury/ http://www.kempruge.com/location/tampa/tampa-personal-injury-legal-attorneys/ Best, Ruben
Intermediate & Advanced SEO | | KempRugeLawGroup0 -
Home Page or Internal Page
I have a website that deals with personalized jewelry, and our main keyword is "Name Necklace".
Intermediate & Advanced SEO | | Tiedemann_Anselm
3 mounth ago i added new page: http://www.onecklace.com/name-necklaces/ And from then google index only this page for my main keyword, and not our home page.
Beacuase the page is new, and we didn't have a lot of link to it, our rank is not so well. I'm considering to remove this page (301 to home page), beacause i think that if google index our home page for this keyword it will be better. I'm not sure if this is a good idea, but i know that our home page have a lot of good links and maybe our rank will be higher. Another thing, because google index this internal page for this keyword, it looks like our home page have no main keyword at all. BTW, before i add this page, google index our main page with this keyword. Please advise... U5S8gyS.png j50XHl4.png0 -
Block search bots on staging server
I want to block bots from all of our client sites on our staging server. Since robots.txt files can easily be copied over when moving a site to production, how can i block bots/crawlers from our staging server (at the server level), but still allow our clients to see/preview their site before launch?
Intermediate & Advanced SEO | | BlueView13010 -
Zip Code Blocks the Search Engines!
I have a site where when you visit the product pages, it asks for your zip code. This is obviously blocking the bots from crawling the site. I know you can basically tell the bots how to ignore the zip code feature but I am not exactly sure how to do this. Any help would be appreciated
Intermediate & Advanced SEO | | lhawk0 -
Should product searches (on site searches) be noindex?
We have a large new site that is suffering from a sitewide panda like penalty. The site has 200k pages indexed by Google. Lots of category and sub category page content and about 25% of the product pages have unique content hand written (vs the other pages using copied content). So it seems our site is labeled as thin. I'm wondering about using noindex paramaters for the internal site search. We have a canonical tag on search results pointing to domain.com/search/ (client thought that would help) but I'm wondering if we need to just no index all the product search results. Thoughts?
Intermediate & Advanced SEO | | iAnalyst.com0 -
Different pages ranking for search terms, often irrelevant.
Website: www.templatemonster.com
Intermediate & Advanced SEO | | templatemonster
Problem: Positions dropped while pages which were ranking previously disappeared from top 100 and now different - often completely irrelevant - pages are ranking. Examples:
Search term: Joomla Templates
Previous Position: 8
Current Position: 35
Previously Ranked Page: http://www.templatemonster.com/joomla-templates.php
Currently Ranked Page: http://www.templatemonster.com/logo-templates.php Similar situation with the following search terms: virtuemart templates, virtuemart themes, prestashop templates, prestashop themes, magento themes, zencart templates, zencart themes, zen cart templates, zen cart themes When: according to the Google Analytics (drop in visitors stats) this happened on July, 2nd Preconditions: we had 45 minutes downtime on July 2-nd - but could this 45 mins have had such disastrous results?
No redirects or canonical URL were used which could lead to such change of ranking page.
No changes in the site's informational structure and design.
In webmaster tools (inbound links report) we saw a website yesterday which had over 800,000 links pointing to our domain - http://moviebestwatch.com/ - and today this site is NOT found in Webmaster Tools report! Also, site is down, domain is quite new (how could it have possibly developed 800,000 pages in such a short time?) and whois is privacy protected. Is this some dirty trick from competitors - could it have possibly influenced our positions? Still, what I completely fail to understand - how could a page like http://www.templatemonster.com/logo-templates.php be the top ranking page for 'Joomla templates' if there is: not a single mention of the word 'Joomla' on the page (or source code), i.e. the page is completely irrelevant to the search term not a single link with 'Joomla templates' anchor text pointing to that page, neither external nor internal PS. No similar changes in other search engines noticed. Also, the pages in question have been re-spidered July 4th and cache shows the right pages, i.e. it is not that Googlebot has seen logotypes page instead of Joomla templates page. I checked any possible reason I could think of (see "Preconditions") but still have no clue - what is going on?1