When You Add a Robots.txt file to a website to block certain URLs, do they disappear from Google's index?
-
I have seen several websites recently that have have far too many webpages indexed by Google, because for each blog post they publish, Google might index the following:
- www.mywebsite.com/blog/title-of-post
- www.mywebsite.com/blog/tag/tag1
- www.mywebsite.com/blog/tag/tag2
- www.mywebsite.com/blog/category/categoryA
- etc
My question is: if you add a robots.txt file that tells Google NOT to index pages in the "tag" and "category" folder, does that mean that the previously indexed pages will eventually disappear from Google's index? Or does it just mean that newly created pages won't get added to the index? Or does it mean nothing at all? thanks for any insight!
-
Hi William
If the pages in question are
- already indexed by Google then if you block them via the robots.txt , they will show up in search result but the meta description will say something along the lines of
A description for this result is not available because of this site's robots.txt – learn more.
2) not indexed by Google for example on a new site , they don't follow it and the pages does not come up in search directly BUT if some external sites link to the pages then they can still come up in the SERP some time down the track.
Your best bet to keep the page out of the public SERP index is the meta robots tag : http://www.robotstxt.org/meta.html
-
William, If the pages in question are linked to from external resources the robots.txt file will not prevent the pages from appearing in the index. Per Moz's Robots.txt and Meta Robots best practices, "the robots.txt tells the engines not to crawl the given URL, but that they may keep the page in the index and display it in in results.
To prevent all robots from indexing a page on your site, place the following meta tag into the section of your page:
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google index new data from my website page
Hi All, We have pages which are created few weeks before hand for Movie reviews in those pages we add value with adding the Movie cast and crew info and what ever info possible before the movie releases. The the movie releases we watch the movies and write reviews which is 500+ words. Now the issue is the pages are indexed a week before... How can i have these review pages scanned immediately when i have the complete review as the review content is not indexed for 3 to 5 days and the first day or 2 is when its important for the reviews to be seen in Google. Regards
On-Page Optimization | | AlexisWithers0 -
Changing existing URL's to improve SEO
Here's a general question: At what PR/page rank (or Moz 'page authority') would you no longer change an existing URL that's cryptic to one that includes the related page keyword or at least relevant terms? Does using a 301 redirect to the new URL preserve the page rank? Thanks!
On-Page Optimization | | flyntime_tx0 -
Google's Page Layout Algorythm
It seems that Google have been or will penalizing websites with too many ads above the fold. Is it me or Google's search result layout is a perfect example of what NOT to do?
On-Page Optimization | | sbrault741 -
Should I buy an established domain that has lost it's high PR due to being offline for several months?
I'm considering purchasing a domain that has sat idle for several months. It was a company's domain that they have owned since the mid1990's but they went out of business. Previously, it had a PR 5 but has since lost it's PR as it has sat 'inactive' with a 'server not found' warning for the past several months. That being said, is there any point in buying the domain (for SEO purposes)? Is there any recourse with Google to try and re-establish the site's credibility or would I be starting over from scratch?
On-Page Optimization | | martybuch230 -
Multiple H1's
Hi, My SEOMOZ report states that I'm using two H1's on most of my pages, for example on this page: http://www.absolutepower.nl/eiwitshakes/proteine-shakes/ I only see one though. Anyone who could clarify this? Thanks! Jasper
On-Page Optimization | | Japking0 -
ECommerce URL's
This is based on a clothing retailer, eCommerce site. In an effort to reduce the length of our product names, we are considering removing terms like long-sleeve, short-sleeve, etc., but leaving that information in the URL. Now, the concern is that we would lose some traction in the SERP's if those descriptive words are left out as the product name is also our page title. Then I think keywords as broad as long-sleeve shirt wouldn't serve us well anyways. One idea we have is that the alt tag on the product image could still display the longer product name that would include long-sleeve, etc. thus having the keyword on the product page. Any ideas or suggestions? Hope this is clear. Seems redundant from a user standpoint to state long-sleeve, etc. in every product name. Thanks - your answers are always so helpful!
On-Page Optimization | | kennyrowe0 -
On-Page SEO Priorities: Title's, Anchor Text or Meta data?
**Any suggestions for prioritized on-page SEO work? Relative weights of importance? ** **What is most important from highest to lowest? ** MetaTag Descriptions? Titles? Anchor Text? Alt Text - for images? Anything else? We might not be able to do everything at once like I desire ......but I do feel we should at least get the ball moving in the right direction. I am looking for ideas or suggestions on what to prioritize for a little bit of on-page SEO work on our website. I personally feel that SEO is pretty important but I am a novice. I have been reading this site the past week and want to convince my webpage guy that on-page SEO is important and that we should at least do a few things and gradually get the work done. Rightfully so our #1 priority is to redesign our landing pages (they are bad) . I also think we should do a little On-Page work concurently. (Lack of on-page SEO is also preventiing me from successfully submitting and being accepted by Dmoz, Yahoo, BOW etc) He is mainly a back engine guy and does a very good job with that. If I were to TELL him to do a few prioritized on-page SEO things what would you suggest? He did do something on the home page at my suggestion but that is all to this point. We have over 400 pages indexed with very little on-page SEO on them. Thank you, UtahTiger
On-Page Optimization | | Boodreaux0 -
Has anyone had experience with the Wix platform and it's SEO qualities?
Wix offers an inexpensive, user friendly platform for building websites. Most of the site is flash, but Wix claims to be SEO friendly. I'm all ears for your feedback and experience with Wix.
On-Page Optimization | | ksracer0