Meta robots at every page rather than using robots.txt for blocking crawlers? How they'll get indexed if we block crawlers?
-
Hi all,
The suggestion to use meta robots tag rather than robots.txt file is to make sure the pages do not get indexed if their hyperlinks are available anywhere on the internet. I don't understand how the pages will be indexed if the entire site is blocked? Even though there are page links are available, will Google really index those pages? One of our site got blocked from robots file but internal links are available on internet for years which are not been indexed. So technically robots.txt file is quite enough right? Please clarify and guide me if I'm wrong.
Thanks
-
I agree with Gaston's approach right up to step 4. If you add the no-indexed pages back into a block in the robots.txt file, you'll end up back where you started from. Because Google will still discover the no-indexed URLs elsewhere and the robots,txt block will stop them from discovering the no-index, and the URLs will likely start to get added to the index again.
No-indexed URLs must not be blocked in robots.txt. Those two processes are mutually exclusive.
-
Hi there,
TLDR; The solution to deindexing and never index again:
- Allow (with robots.txt) the web to be crawable
- Aplly meta robots tag: noindex,follow
- Wait somte weeks to be completely deindexed
- block the entire site/section with robots.txt
Robots.txt and the robots meta tag can make the same effect, but to understand them must be analyzed separatedly.
-
Robots.txt, here you just tell bots where they can go BEFORE they crawl any of the website. This is just a signal, not a directive... Because robots can choose to ignore the what's in the file. Here you can block from the entire web, to an entire section or just specific pages. More info: Robots.txt official page and a really cool and complete guide to robots.txt
-
Robots meta tag, with it you have more signals to tell, the most used are: noindex, nofollow and follow, due to the usual issues about indexing. More info: Robots.txt offical page, Google developers, Meta Robots directive - Moz and a complete guide to meta robots tag - YOAST.
Hope this is what you wanted.
Best luck
GR.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
AMP pages - should we be creating AMP versions of all site pages?
Hi all, Just wondering what people's opinions are on AMP pages - having seen the Google demo of how AMP pages will be given visibility on page one of Google for news-based content, do you think it is worth considering creating AMP versions of all pages, ready for when Google expands its inclusion of these super-fast pages?
Algorithm Updates | | A_Q1 -
Thoughts on Google's Autocomplete hurting organic SEO?
A client sent over an article about how Google's Autocomplete eliminates your chance for clicks. Saying that if your competitor is higher than you, the user will bypass the page one organic rank and click on a specific business from the autocomplete which in turn presents an entire page one result for that business. So in a sense they are wondering why they're doing organic SEO if potential customers are just going to bypass the page one organic results. I would love to hear thoughts from like minded people on this as I have to start proving my case with articles, facts, data, and research.
Algorithm Updates | | MERGE-Chicago0 -
How to you get a company details to appear on the right side of google searches? For example is you type dr martens in google , the search page shows links and then a snippet of the company on the right. Thank you.
For example is you type dr martens in google , the search page shows links and then a snippet of the company on the right. Thank you. https://www.google.com/search?q=dr+martens&oq=dr+martens&aqs=chrome..69i57j0l5.2176j0j7&sourceid=chrome&espv=210&es_sm=122&ie=UTF-8
Algorithm Updates | | vijayvasu0 -
SERPS appearing and vanishing every 15 days
I have a regional careers website on a small seo retainer contract. I manage the google webmaster account and use moz tools to help keep pages finely tuned. The site has been experiencing a phenomena now for about 6 months whereby the SERPS (we work with around 10 keyphrases) are placed in top 5 positions and generate healthy google traffic and will then suddenly vanish. The cycle is roughly 1 - 2 weeks top page, followed by 2 - 3 weeks outside the top 5 pages of google results. I contacted Google via webmaster central to ask for help and they simply confirmed that there were no penalties being applied and it was normal algorithm updates. I manage the seo for 20+ sites and have not seen this phenomenon elsewhere, not since the days of 'google dance' years ago. Has anyone else had this experience ? The SERPS always come back after a couple of weeks but then vanish again after a week or so this cycle has been ongoing for 6 months now and we have eliminated all symptoms we can think of. Your feedback would be much appreciated in this matter.
Algorithm Updates | | broadshout0 -
Meta Keywords ... Client request
I have a client who would like to include Meta Keywords in their site as they have seen competitors with it, we haven't used Meta Keywords tag before as doesn't impact ranking as we all know - however if I am to put them in - is there a limit - would 200 + negatively impact the site? Thanks
Algorithm Updates | | lauratagdigital0 -
Google's not indexing my blog posts anymore! Why?
Google just recently stopped indexing my blog posts immediately after being published, why could this be? I would usually post a blog post and it would be in google results within 45 seconds, now they don't show up until 6 hours later, if at all (a few never even showed up). Also, my home page doesn't even refresh when I make a change to the site. My site is CantStopHipHop [dot] comI have all in one SEO, xml sitemap generator, and webmaster tools and nothing seemed irregular in the settings.I appreciate any thoughts/help/suggestions.
Algorithm Updates | | bb2550 -
How do I get the expanded results in a Google search?
I notice for certain site (ex: mint.com) that when I search, the top result has a very detailed view with options to click to different subsections of the site. However for my site, even though we're consistently the top result for our branded terms, the result is still only a single line item. How do I adjust this?
Algorithm Updates | | syount1 -
Accidently blocked our site for an evening?
Yesterday at about 5pm I switched our site to a new server and accidentally blocked our site from google for the evening. our domain is posnation.com and we are ranked in the top 3 in almost all pos related keywords. When i got in this morning i realized the mistake and went to google web tools and noticed the site was blocked so i went to fetch as google bot and corrected that. Now the message says: Check to see that your robots.txt is working as expected. (Any changes you make to the robots.txt content below will not be saved.)
Algorithm Updates | | POSNation
robots.txt file Downloaded Status
http://www.posnation.com/robots.txt 1 hours ago 200 (Success) When you go to google and type "pos systems" we are still #2 so i assume all is still ok. My question is will this potentially hurt our rankings and should i be worried and is there anything else I can do.0