When You Add a Robots.txt file to a website to block certain URLs, do they disappear from Google's index?
-
I have seen several websites recently that have have far too many webpages indexed by Google, because for each blog post they publish, Google might index the following:
- www.mywebsite.com/blog/title-of-post
- www.mywebsite.com/blog/tag/tag1
- www.mywebsite.com/blog/tag/tag2
- www.mywebsite.com/blog/category/categoryA
- etc
My question is: if you add a robots.txt file that tells Google NOT to index pages in the "tag" and "category" folder, does that mean that the previously indexed pages will eventually disappear from Google's index? Or does it just mean that newly created pages won't get added to the index? Or does it mean nothing at all? thanks for any insight!
-
Hi William
If the pages in question are
- already indexed by Google then if you block them via the robots.txt , they will show up in search result but the meta description will say something along the lines of
A description for this result is not available because of this site's robots.txt – learn more.
2) not indexed by Google for example on a new site , they don't follow it and the pages does not come up in search directly BUT if some external sites link to the pages then they can still come up in the SERP some time down the track.
Your best bet to keep the page out of the public SERP index is the meta robots tag : http://www.robotstxt.org/meta.html
-
William, If the pages in question are linked to from external resources the robots.txt file will not prevent the pages from appearing in the index. Per Moz's Robots.txt and Meta Robots best practices, "the robots.txt tells the engines not to crawl the given URL, but that they may keep the page in the index and display it in in results.
To prevent all robots from indexing a page on your site, place the following meta tag into the section of your page:
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
No index for http version of website
Hi, I've had a message from Google search console to say the sitemap for the http version of my site is tagged as no index. As the https version is indexed, do I need to change the http version to be indexed as well? Do I need to keep the http version of the site in search console alongside the https version, or should I remove it? Advice appreciated!
On-Page Optimization | | Robingoodlad0 -
Number of internal links and passing 'link juice' down to key pages.
Howdy Moz friends. I've just been checking out this post on Moz from 2011 and wanted to know how relevant it is today? I'm particularly interested in a number of links we have on our HP potentially harming important landing page rankings because not enough 'link juice is getting to them i.e) are they are being diluted by all the many other links on the page? (deeper pages, faqs, etc etc) It seems strange to me that as Google as has got more sophisticated this would still be that relevant (thus the reason for posting). Anyway, I thought I was definitely worth asking. If we can leverage more out of our on-page efforts then great 🙂
On-Page Optimization | | isaac6630 -
No-index all the posts of a category
Hi everyone! I would like no-indexing all the posts of a specific category of my wordpress site. The problem is that the structure of my URL is composed without /category/: www.site-name.ext/date/post-name/
On-Page Optimization | | salvyy
so without /category-name/ Is possibile to disallow the indexing of all the posts of the category via robots.txt? Using Yoast Plugin I can put the no-index for each post, but I would like to put the no-index (or disallow/) a time for all the post of the category. Thanks in advance for your help and sorry for my english. Mike0 -
Google ranking is HORRIBLE. Following SEOMoz suggestions and just can't climb.
First of all, the URL is stores.dhsequipment.com. In January, this online store switched from Homestead to Big Commerce. Since the store updated, we decided now is the time to update our product descriptions, URL's, title tags and meta descriptions. (For the first time, we had the ability to customize our URL's.) Product Description: I went through 2,500 products and updated the product description. I added an H1 & H2 to each description, and included pertinent information such as part numbers. Each product also received a new page title, meta description (which is usually the first line of the product description, don't know if this is bad or not) and a new URL, (which did redirect). Once I would complete a section, I would submit a new sitemap to Webmaster Tools. After a month and nothing happening, I started using SEOMoz which helped me rebuild some of my more important pages, such as the home page and main category pages such as: http://stores.dhsequipmentparts.com/stihl-ts420-parts/
On-Page Optimization | | pearldesign
http://stores.dhsequipmentparts.com/stihl-ts700-parts-stihl-ts800-parts/ I fetched these pages in Webmaster Tools after completion. However, it's been several weeks since and I'm still on page 4 or 5 in the SERPs. Just a little history on the store; this store has been in operation for more than 6 years. Previously, we ranked on page one for 75%+ of our products. My belief is because our URL's had history, probably more so than our competitors. I'm not sure what I should do. Business is super slow and we can't afford to wait much longer.0 -
If I put 'keyword/url' combination to 'stop run weekly', will it dissapear from the summary page in the on-page grader?
The summary page of the on-page grader chooses the keyword and url combination itself. Now if I choose another combination, I would like the former to dissapear from the summary page. The only option is 'stop running weekly'. But will it disappear from the list also?
On-Page Optimization | | jongeneelbv0 -
What's a reasonable bounce rate for school website?
Does anyone have a baseline on what the average bounce rate should be on a school website?
On-Page Optimization | | BillyBobGriffin0 -
Canonical URLs and SEO
After publishing my new wordpress website my on page analysis shows two canonical urls on my homepage. In checking the source code the Yoast SEO plugin states my canoncial url is www.homepage.ca/ This is the only one showing. Having said this I do not add the trailing slash to anything on my url structure. Could this be why the on page analysis tool says there are 2? If so is there any way to resolve this?
On-Page Optimization | | casper4340 -
Canonical URL's - Fixed but still negatively impacted
I recently noticed that our canonical url's were not set up correctly. The incorrect setup predates me but it could have been in place for close to a year, maybe a bit more. Each of the url's had a "sortby" parameter on all of them. I had our platform provider make the fix and now everything is as it should be. I do see issues caused by this in Google Webmaster, for instance in the HTML suggestions it's telling me that pages have duplicate title tags when in fact this is the same page but with a variety of url parameters at the end of the url. To me this just highlights that there is a problem and we are being negatively impacted by the previous implementation. My question is has anyone been in this situation? Is there any way to flush this out or push Google to relook at this? Or is this a sit and be patient situation. I'm also slightly curious if Google will at some point look and see that the canonical urls were changed and then throw up a red flag even though they are finally the way they should be. Any feedback is appreciated. Thanks,
On-Page Optimization | | dgmiles
Dave0