When You Add a Robots.txt file to a website to block certain URLs, do they disappear from Google's index?
-
I have seen several websites recently that have have far too many webpages indexed by Google, because for each blog post they publish, Google might index the following:
- www.mywebsite.com/blog/title-of-post
- www.mywebsite.com/blog/tag/tag1
- www.mywebsite.com/blog/tag/tag2
- www.mywebsite.com/blog/category/categoryA
- etc
My question is: if you add a robots.txt file that tells Google NOT to index pages in the "tag" and "category" folder, does that mean that the previously indexed pages will eventually disappear from Google's index? Or does it just mean that newly created pages won't get added to the index? Or does it mean nothing at all? thanks for any insight!
-
Hi William
If the pages in question are
- already indexed by Google then if you block them via the robots.txt , they will show up in search result but the meta description will say something along the lines of
A description for this result is not available because of this site's robots.txt – learn more.
2) not indexed by Google for example on a new site , they don't follow it and the pages does not come up in search directly BUT if some external sites link to the pages then they can still come up in the SERP some time down the track.
Your best bet to keep the page out of the public SERP index is the meta robots tag : http://www.robotstxt.org/meta.html
-
William, If the pages in question are linked to from external resources the robots.txt file will not prevent the pages from appearing in the index. Per Moz's Robots.txt and Meta Robots best practices, "the robots.txt tells the engines not to crawl the given URL, but that they may keep the page in the index and display it in in results.
To prevent all robots from indexing a page on your site, place the following meta tag into the section of your page:
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
No index, or no index no follow?
Wondering if I could garner some views on this issue please. I'm about to add an affiliate store to a website I own, the site has a couple of pages of unique content (blogs, articles, advice etc on home improvement - all written by my team). Obviously, the affiliate store will not be unique content, it will be made using the datafeeds from cj.com et al, and so I don't want to get any duplicate content type penalties from Google for this store. Should I add a no index to the pages and allow the bots to still crawl them, or should I add no index and no follow? Ideally I would like to get the affiliate store category pages indexed as they will be a mixture of lots of different merchants and be fairly unique. Can Google still mark the site down for duplicate content if it can crawl it, even if it is noindex? Thanks, Carl
On-Page Optimization | | Grumpy_Carl0 -
How to website faster?
my website's speed is slow, and in my source code website have many line css and javascript from themes and plugin when install Can I @import other css file to one file ? and if can, how to? my site is : itvseo.com, i using wordpress themes from rockketthemes, it has many plugin (rokbox, rok seach, gantry...) --> so site load slow --> how can fix it see images below Thank very much dgox.png
On-Page Optimization | | ITVSEO.COM0 -
Website redesign and it's impact on ranking
Hey Everyone, I have had a website in place for over 7 years and I am now at the point where I need to implement a redesign in order to sell our product more effectively. -URL's will stay the same
On-Page Optimization | | Justin45
-Content will change but be very similar page-to-page
-Title & Meta tags will remain the same I'm planning on taking the site from a non-backend site to a wordpress site so the navigation will change. Does anyone foresee that this change will have any dramatic effect on site ranking? Thank You!0 -
Does Google follow link path or url path when it comes to passing link juice
I noticed something with one of my sites and now I am thinking I made a boo boo (I think) here is what I have On my homepage I have 5 links Link1
On-Page Optimization | | cbielich
Link2
Link3
Link4
Link5 Links 1 - 4 go to a page and stops there. So my URL structure is www.mydomain.com/Link1
www.mydomain.com/Link2
www.mydomain.com/Link3
www.mydomain.com/Link4 So naturally my link juice passes down to these links evenly. Link5 also goes to another page, but on that page I have more links that go down further. www.mydomain.com/Link5 -> more links On page Link5 I have links that go to more pages, BUT my URL structure for these pages go like this Lets say on Link5 page I have another link that goes to AnotherLink1, AnotherLink2 and AnotherLink3 When you click on those links it takes you to those pages just fine, BUT my URL structure is like this www.mydomain.com/AnotherLink1
www.mydomain.com/AnotherLink2
www.mydomain.com/AnotherLink3 Basically I put all the "AnotherLink1-3" in the root directory as well. My question is concerning how Google passes the link Juice from my pages and if it is passing based on the path of the links and how they point to those pages, or do they pass link juice based on the URL structure. So since "AnotherLink1-3" is located in the root directory am I dividing my link juice from my home page to all the links as well based on the URL structure. For instance www.mydomain.com/Link1
www.mydomain.com/Link2
www.mydomain.com/Link3
www.mydomain.com/Link4
www.mydomain.com/Link5
www.mydomain.com/AnotherLink1
www.mydomain.com/AnotherLink2
www.mydomain.com/AnotherLink3 Do I need to change my path for Link5 page to www.mydomain.com/Link5/AnotherLink1
www.mydomain.com/Link5/AnotherLink2
www.mydomain.com/Link5/AnotherLink3 ?0 -
Client has 2 websites... won't do away with old site.
Ok my situation is: We have built a new site for a client and they would like to keep both sites working. Should I just attempt to optimize both sites to rank well in their area? Or should I be suggesting a 301 redirect from the old domain to the new? Other options? Thanks in advance
On-Page Optimization | | IntermarkGroup0 -
Removing old URLs from Google
Hello, I am sure that this question has been asked many times, but I am still not sure what to do about the following: Our site's URL structure has changed a few times in the past few months. Recenty, we have changed our URLs to become more SEO friendly. However, Google has indexed the old URLs as well. To give an example: The following page in our website shows the following URLs in Google Webmaster Tools: Confúcio e Seus Ensinamentos /artigo/68_38/2/as_religioes_iv_confucio_e_seus_ensinamentos//aula/14_6132/vestibular/confucio_e_seus_ensinamentos//aula/1_14_6132/vestibular/confucio_e_seus_ensinamentos//aula/_14_6132/Vestibular/confucio_e_seus_ensinamentos//aula/ensino/confucio_e_seus_ensinamentos/ The correct URL is the last one. What should I do about the other ones? Almost all the pages in our website have this problem. We have redirected the old URLs to the new ones, but is there anything else we should do? We were asking Google to remove them, but Google has informed us that it has reached the limit. Please advise us on waht we should do. We have removed the old sitemap with the old URLs. What else must we do? Thank you very much.
On-Page Optimization | | Tev0 -
How long after a URL starts showing a 404 does Google stop crawling?
Before hiring me to do SEO, a client re-launched their site and did not 301 the old URLs to the new. Only the home page URL stayed the same. For a month after the re-launch, the old URLs returned a 404. For the next month, all 404 pages (basically any non-existent URL) were 301'd to the home page. Finally, 2 months after launching, they properly 301'd the old URLs to the new. Now, the new URLs are not ranking well. I assume it's too late to realize any benefit from the 301's, just checking to see if anybody has any insight into how long Google keeps trying to crawl old/404/improperly 301'd URLs. Thanks!
On-Page Optimization | | AndrewMiller0 -
How to Resolve Google Crawling Issues for My eCommerce Website?
I want to resolve Google crawling issues for my eCommerce website. My website is as follow. http://www.vistastores.com/ Google have crawled only 97 webpages from my website. My website is quite old. (~More than 6 months) But, Google have indexed only 97 webpages. I have created one campaign over SEOmoz tool and found some errors over there. So, I just assumed that due to it Google did not crawled my website. But, I have created one another campaign for my competitor website to know actual status and reason behind it. I found that, my competitor website have more error compare to me but, Google have crawled maximum pages compare to me. So, What is reason behind it? How can I improve my crawling rate and index maximum webpages to Google? [6133009604_af85d29730_b.jpg](img src=) 6133009604_af85d29730_b.jpg 6133009604_af85d29730_b.jpg 6139706697_4e252fdb82_b.jpg
On-Page Optimization | | CommercePundit0