Meta robots at every page rather than using robots.txt for blocking crawlers? How they'll get indexed if we block crawlers?
-
Hi all,
The suggestion to use meta robots tag rather than robots.txt file is to make sure the pages do not get indexed if their hyperlinks are available anywhere on the internet. I don't understand how the pages will be indexed if the entire site is blocked? Even though there are page links are available, will Google really index those pages? One of our site got blocked from robots file but internal links are available on internet for years which are not been indexed. So technically robots.txt file is quite enough right? Please clarify and guide me if I'm wrong.
Thanks
-
I agree with Gaston's approach right up to step 4. If you add the no-indexed pages back into a block in the robots.txt file, you'll end up back where you started from. Because Google will still discover the no-indexed URLs elsewhere and the robots,txt block will stop them from discovering the no-index, and the URLs will likely start to get added to the index again.
No-indexed URLs must not be blocked in robots.txt. Those two processes are mutually exclusive.
-
Hi there,
TLDR; The solution to deindexing and never index again:
- Allow (with robots.txt) the web to be crawable
- Aplly meta robots tag: noindex,follow
- Wait somte weeks to be completely deindexed
- block the entire site/section with robots.txt
Robots.txt and the robots meta tag can make the same effect, but to understand them must be analyzed separatedly.
-
Robots.txt, here you just tell bots where they can go BEFORE they crawl any of the website. This is just a signal, not a directive... Because robots can choose to ignore the what's in the file. Here you can block from the entire web, to an entire section or just specific pages. More info: Robots.txt official page and a really cool and complete guide to robots.txt
-
Robots meta tag, with it you have more signals to tell, the most used are: noindex, nofollow and follow, due to the usual issues about indexing. More info: Robots.txt offical page, Google developers, Meta Robots directive - Moz and a complete guide to meta robots tag - YOAST.
Hope this is what you wanted.
Best luck
GR.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Our sitemap is not indexed i Google even though it's successfully processed
Hi, Ours is a WP hosted website. We have submitted the XML sitemap with a WP plugin. It's been successfully processed by Google but it's not been indexed in and can't be found in SERP. How to get this indexed? Will there be any low crawling of sitemap as it's not indexed? Thanks
Algorithm Updates | | vtmoz0 -
Do we need to maintain consistency in page titles suffix?
Hi all, We usually give "brand & primary keyword" across all pages in website like "vertigo tiles". Do we need to maintain this suffix across all page titles? What if we change according to the page? Will Google downlook for not maintaining these page titles suffix like I mentioned? Thanks
Algorithm Updates | | vtmoz0 -
Lots of dublicate titles and pages on search page
I own a paiting website with a lot of searchable paintings. The "search paintings" feature creates tons of dublicate pages and titles. See here:
Algorithm Updates | | KasperGJ
http://www.maleribasen.dk/soegmaleri.asp I guess the problem is, that the URL can actually be different and still return the same content. First time you click the "Search paintings" the URL will shown as above. But as soon as users
begin to definere they search to the left and use the "Search button" the top URL changes. So, depending on how the top URL looks different results are shown. This is pretty standard in searches. But it returns tons of dublicate pages and titles. How, do you guys cope with that? Is there a clever way to use ref="cannonical" or some other smart way to avoid this? /Kasper0 -
On-page Optimization
Hi, I have two campaigns and neither have any statistics for on-page optimization. Am I doing something wrong or how do I make these stats appear? I would like to improve my website. Thank you in advanced for any pointers or shared experience you may give me!
Algorithm Updates | | Pixeltistic0 -
Can Google display a diffrent page title?
Hi if I search google UK for the phrase car leasing, google returns my listing as Car Lease Deals However the same search on Yahoo or Bing bring back Contract Hire | Vehicle & Car Leasing Deals | Car Lease Deals this is the real page title. Why would this happen? Thanks Andy
Algorithm Updates | | First-VehicleLeasing0 -
Should I block non-informative pages from Google's index?
Our site has about 1000 pages indexed, and the vast majority of them are not useful, and/or contain little content. Some of these are: -Galleries
Algorithm Updates | | UnderRugSwept
-Pages of images with no text except for navigation
-Popup windows that contain further information about something but contain no navigation, and sometimes only a couple sentences My question is whether or not I should put a noindex in the meta tags. I think it would be good because the ratio of quality to low quality pages right now is not good at all. I am apprehensive because if I'm blocking more than half my site from Google, won't Google see that as a suspicious or bad practice?1 -
Using Anchor Text to help with search engines
Hi i am wanting my site to come up in the search engines under certain search words such as Hypno Band but i do not want the words displayed on my site. I would like to know if there is a way to let the search engines know that the page is relevant to the chosen keywords without having them on my site. I have been reading about anchor text and would like to know the best way to use these and the best way to let the search engines know that the page is rellevant to the keywords. any help would be great
Algorithm Updates | | ClaireH-1848860 -
Why google index ip address instead of the domain name?
I have a website ,now google index ip address of it instead of the domain name,I have used 301 redirected to the domain name,but how to change the index IP to its domain name? And why google index the IP address?
Algorithm Updates | | frankfans1170