When You Add a Robots.txt file to a website to block certain URLs, do they disappear from Google's index?

[williammarlow]

I have seen several websites recently that have have far too many webpages indexed by Google, because for each blog post they publish, Google might index the following:

• www.mywebsite.com/blog/title-of-post
• www.mywebsite.com/blog/tag/tag1
• www.mywebsite.com/blog/tag/tag2
• www.mywebsite.com/blog/category/categoryA
• etc

My question is: if you add a robots.txt file that tells Google NOT to index pages in the "tag" and "category" folder, does that mean that the previously indexed pages will eventually disappear from Google's index?  Or does it just mean that newly created pages won't get added to the index?  Or does it mean nothing at all?  thanks for any insight!

[Saijo.George]

Hi  William

If the pages in question are

1. already indexed by Google then if you block them via the robots.txt , they will show up in search result but the meta description will say something along the lines of

A description for this result is not available because of this site's robots.txt – learn more.

2) not indexed by Google for example on a new site , they don't follow it and the pages does not come up in search directly BUT if some external sites link to the pages then they can still come up in the SERP some time down the track.

Your best bet to keep the page out of the public SERP index is the meta robots tag : http://www.robotstxt.org/meta.html

[Chris.Menke]

William, If the pages in question are linked to from external resources the robots.txt file will not prevent  the pages from appearing in the index. Per Moz's Robots.txt and Meta Robots best practices, "the robots.txt tells the engines not to crawl the given URL, but that they may keep the page in the index and display it in in results.

To prevent all robots from indexing a page on your site, place the following meta tag into the section of your page:

https://support.google.com/webmasters/answer/93710?hl=en