When You Add a Robots.txt file to a website to block certain URLs, do they disappear from Google's index?
-
I have seen several websites recently that have have far too many webpages indexed by Google, because for each blog post they publish, Google might index the following:
- www.mywebsite.com/blog/title-of-post
- www.mywebsite.com/blog/tag/tag1
- www.mywebsite.com/blog/tag/tag2
- www.mywebsite.com/blog/category/categoryA
- etc
My question is: if you add a robots.txt file that tells Google NOT to index pages in the "tag" and "category" folder, does that mean that the previously indexed pages will eventually disappear from Google's index? Or does it just mean that newly created pages won't get added to the index? Or does it mean nothing at all? thanks for any insight!
-
Hi William
If the pages in question are
- already indexed by Google then if you block them via the robots.txt , they will show up in search result but the meta description will say something along the lines of
A description for this result is not available because of this site's robots.txt – learn more.
2) not indexed by Google for example on a new site , they don't follow it and the pages does not come up in search directly BUT if some external sites link to the pages then they can still come up in the SERP some time down the track.
Your best bet to keep the page out of the public SERP index is the meta robots tag : http://www.robotstxt.org/meta.html
-
William, If the pages in question are linked to from external resources the robots.txt file will not prevent the pages from appearing in the index. Per Moz's Robots.txt and Meta Robots best practices, "the robots.txt tells the engines not to crawl the given URL, but that they may keep the page in the index and display it in in results.
To prevent all robots from indexing a page on your site, place the following meta tag into the section of your page:
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
For an e-commerce product category page that has several funnels to specific products, for SEO purposes does it matter whether the category page's overview content is above or below those funnels?
We manage an e-commerce site. On a category page, there are several funnels to specific products. We moved the category overview content below those funnels to make it easier for users to quickly get to products. Seems more user friendly to me, but could that move of the main content to the lower part of the page be a negative ranking factor?
On-Page Optimization | | PKI_Niles0 -
Toxic URL???
Hi I have a URL that produced page 1, number 1 to 3 for most of our industries top phrases. Then we received a google penalty, (as did several of our competitors on the same day). We were effectively wiped from google. After much disavowing we were allowed back into the search results, this took about 3 months. I have employed the services of a top London SEO company for over a year now and have seen no significant improvement. I believe they are doing there best, however there results are VERY poor. According to the various tools, (searchmetrics, woorank, semrush) to name but a few, our site scores very well, yet we are not getting the results. Page one seems to be full of totally new websites, most of which I have never heard of, and have appeared from nowhere. Should I scrap our URL and put up a completely new one, and put a redirect from the original one? This would be a biggy since our url has been around for 20 years. Thanks for reading. Andy
On-Page Optimization | | First-VehicleLeasing0 -
Does this index well
hi i have been looking at this template but as the content will be generated from their database will it index well - am i better to build a static equivalent using something like visual composer as i am not that technical: http://realhomes.inspirythemes.biz/listing/
On-Page Optimization | | neilhenderson0 -
Google's mobile-friendly update. How significant is the impact for us?
Hi guys. Recently I got an email from Webmaster-tools saying our site is poorly optimised for mobile devices, and that it’s going to heavily affect rankings from April 21st. I’m worried to say the least. We literary cannot afford a hit on traffic at the moment 😞 We rank well for niche terms like ‘customised diary’ and ‘personalised diary’. So question... Because we rank well for these very specific searches will we still take a hit on rankings after the update? Won’t our high relevancy for those search terms be enough to keep us high in the results? Also, do you know if this change is specific to the users device? E.g) Someone on a mobile device will get mobile-friendly results, whilst users on a laptop will get different results altogether? I'm just trying to get a sense of how much this update will effect us. Any isights, suggestion, or thoughts would be greatly appreciated. Our site. Thanks in advance. This community is invaluable to us 🙂 Isaac - TOAD Diaries.
On-Page Optimization | | isaac6630 -
Why Can't I Get Indexed?
I cannot seem to get my website indexed by Google! I submitted the sitemap using Google WMT about a month ago but only one page is being indexed. There are very few backlinks to the site, so I don't believe there are any penalties due to over-optimization that would prevent indexing. Also, my robots.txt file is properly configured and is not preventing any pages from being crawled. I've tried using the "Fetch as Google" settings in WMT with no luck. Any ideas?
On-Page Optimization | | socialfirestarter0 -
Description tag not showing in the SERPs because page is blocked by Robots, but the page isn't blocked. Any help?
While checking some SERP results for a few pages of a site this morning I noticed that some pages were returning this message instead of a description tag, A description for this result is not avaliable because of this site's robot.s.txt The odd thing is the page isn't blocked in the Robots.txt. The page is using Yoast SEO Plugin to populate meta data though. Anyone else had this happen and have a fix?
On-Page Optimization | | mac22330 -
Is Googlebot seeing text in my 'Read More' bar and could they even be penalising me for it?
Hi guys. Our web developers have put a read more bar that contains the on page SEO text for our webpage. By default, the read more bar is not expanded and you cannot see the text within. If you click 'Read More', the box expands and the text is shown on the page. I was wondering if googlebot is seeing this text at all - it's really important that it does because it contains all the on-page SEO text. I also wondered if this type of approach is still considered 'white hat'? If it's not, could google be penalising us for it? Thanks,
On-Page Optimization | | CarlDarby0 -
Robots.txt file
Does it serve any purpose if we omit robots.txt file ? I wonder if spider has to read all the pages, why do we insert robots.txt file ?
On-Page Optimization | | seoug_20050