When You Add a Robots.txt file to a website to block certain URLs, do they disappear from Google's index?
-
I have seen several websites recently that have have far too many webpages indexed by Google, because for each blog post they publish, Google might index the following:
- www.mywebsite.com/blog/title-of-post
- www.mywebsite.com/blog/tag/tag1
- www.mywebsite.com/blog/tag/tag2
- www.mywebsite.com/blog/category/categoryA
- etc
My question is: if you add a robots.txt file that tells Google NOT to index pages in the "tag" and "category" folder, does that mean that the previously indexed pages will eventually disappear from Google's index? Or does it just mean that newly created pages won't get added to the index? Or does it mean nothing at all? thanks for any insight!
-
Hi William
If the pages in question are
- already indexed by Google then if you block them via the robots.txt , they will show up in search result but the meta description will say something along the lines of
A description for this result is not available because of this site's robots.txt – learn more.
2) not indexed by Google for example on a new site , they don't follow it and the pages does not come up in search directly BUT if some external sites link to the pages then they can still come up in the SERP some time down the track.
Your best bet to keep the page out of the public SERP index is the meta robots tag : http://www.robotstxt.org/meta.html
-
William, If the pages in question are linked to from external resources the robots.txt file will not prevent the pages from appearing in the index. Per Moz's Robots.txt and Meta Robots best practices, "the robots.txt tells the engines not to crawl the given URL, but that they may keep the page in the index and display it in in results.
To prevent all robots from indexing a page on your site, place the following meta tag into the section of your page:
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How do I redirect SEO from pages on old website to new website with new domain name?
How do I redirect SEO from pages on old website to new website with new domain name? My website in squarespace has good rankings and we have rebranded to a new name and enw domain name, how do I move across the rankings fomr the old site in squarespace to my new site thats in wordpress please?
On-Page Optimization | | Carrotpower0 -
Why is Moz's Schema so bare
I use to use Moz as an example by looking at their site using Google's Rich snippet tool, Today I checked and it is so bare with errors https://search.google.com/structured-data/testing-tool/u/0/#url=moz.com What happen?Why the sudden change?
On-Page Optimization | | uBreakiFix0 -
How can you activate the 'Results From' internal search bar on Google SERP?
Hi There, I am hoping someone can advise me on getting the 'Results From' sitelink to display for my site on the Google SERP? I have searched far and wide for the answer with no luck. I'd really appreciate your advice. Thanks! Internal_Search_Google_SERP_zps75a5383e.jpg
On-Page Optimization | | tmg.seo0 -
Will Google penalise me for duplicating my own website on a new domain?
Hi guys, Currently, we are using Magento Enterprise to run two separate ecommerce websites, for arguments sake lets say these two websites are ElectronicsX.com and electronicsY.co.uk. ElectronicsX.com isn't doing too well, it has little traffic, makes next to nothing each month and is just a waste of resources. ElectronicsY.co.uk on the other hand is getting 30,000 unique visits every month and is over exceeding every month in terms of revenue in sales. At the moment, the only traffic we get from Europe tends to be the odd "word of mouth" customer and English customers who have emigrated to Europe but still like to use English companies. We can't rank on Google FR or ES or anything like that because the website we have "electronicsY" is on a .co.uk TLD (a GeoTLD). So what we want to do is take ALL the content from electronicsY.co.uk and place it on electronicsX.com so that we can start targeting Europe and ranking in the serps internationally. What sort of effect will that have in terms of penalisation from Google? Or because the websites are on the same C block, would it have any effect at all? Thanks guys Tom
On-Page Optimization | | tomhall900 -
Url structure
Hi Guys, Wondering what is better for url structure say for example a key word "slow cooker" example.com/slowcooker or example.com/slow-cooker ? Thank you 🙂
On-Page Optimization | | GetApp0 -
Google Doesn't Display A Right Page Title
For some reason Google Displays a wrong page title of some of my pages. E.g. page http://www.imoney.my/home-loan The title in the search reach results says "Home Loan - iMoney", but the one I've set up is <title></span><span class="webkit-html-tag">Housing Loan: Compare Mortgages of All Malaysian Banks @iMoney.my</span><span class="webkit-html-tag"></title> Even when I preview it on the preview tool, it shows the full title, but when I google - again the short one. Does anyone know what the reason for that is?
On-Page Optimization | | imoney0 -
Does google treat all urls equal?
Sorry for the lame title, i couldn't think of a better one. I want to know if google treats this: http://www.domain.com/products/some-product-name the same as it would treat: http://www.domain.com/?products=some-product-name if not, could you tell me the differences?
On-Page Optimization | | adriandg0 -
Value of PDF's in SEO
I have a client who has a lot of information in PDF form. They think they should move some of it over into HTML pages so it indexes better. Is there a benefit to converting these PDF's into HTML pages? It seems to me that HTML pages would be good, IF they are relevant pages that could be used online.
On-Page Optimization | | lvstrickland0