Meta robots at every page rather than using robots.txt for blocking crawlers? How they'll get indexed if we block crawlers?
-
Hi all,
The suggestion to use meta robots tag rather than robots.txt file is to make sure the pages do not get indexed if their hyperlinks are available anywhere on the internet. I don't understand how the pages will be indexed if the entire site is blocked? Even though there are page links are available, will Google really index those pages? One of our site got blocked from robots file but internal links are available on internet for years which are not been indexed. So technically robots.txt file is quite enough right? Please clarify and guide me if I'm wrong.
Thanks
-
I agree with Gaston's approach right up to step 4. If you add the no-indexed pages back into a block in the robots.txt file, you'll end up back where you started from. Because Google will still discover the no-indexed URLs elsewhere and the robots,txt block will stop them from discovering the no-index, and the URLs will likely start to get added to the index again.
No-indexed URLs must not be blocked in robots.txt. Those two processes are mutually exclusive.
-
Hi there,
TLDR; The solution to deindexing and never index again:
- Allow (with robots.txt) the web to be crawable
- Aplly meta robots tag: noindex,follow
- Wait somte weeks to be completely deindexed
- block the entire site/section with robots.txt
Robots.txt and the robots meta tag can make the same effect, but to understand them must be analyzed separatedly.
-
Robots.txt, here you just tell bots where they can go BEFORE they crawl any of the website. This is just a signal, not a directive... Because robots can choose to ignore the what's in the file. Here you can block from the entire web, to an entire section or just specific pages. More info: Robots.txt official page and a really cool and complete guide to robots.txt
-
Robots meta tag, with it you have more signals to tell, the most used are: noindex, nofollow and follow, due to the usual issues about indexing. More info: Robots.txt offical page, Google developers, Meta Robots directive - Moz and a complete guide to meta robots tag - YOAST.
Hope this is what you wanted.
Best luck
GR.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Do keyword target landing pages increase rankings?
Let's say we create landing pages for targeted keywords in our niche. So like we have landing pages optimised for 80% of the top keywords with decent search volume. If these pages started ranking at first page or around; will this scenario improves the ranking of website? Right now, only few of our top pages are ranking good. Planning to create more of such.
Algorithm Updates | | vtmoz0 -
Landing page redirect along with complete content
Hi Moz community, We have a page with "keyword" we are targeting in slug like website.com/keyword/. This page doesn't have much back-links or visits like homepage. So we decided to redirect homepage to /keyword page along with complete content. Will this going to hurt? Only change anybody can notice is URL. Are there any risks involved. I think this is the best way to highlight the page we been thinking about. Thanks
Algorithm Updates | | vtmoz0 -
How to find keywords competitor is using
I am doing work for a landscaping company and having trouble with finding the best keywords. Most of keywords are so expensive on adwords to use, so obviously we want to optimize as best possible. How do I find what keywords competitors are using for campaigns and/or optimized for? thx.
Algorithm Updates | | SexyLeggings0 -
Can a google data refresh knock your pages out of the rankings?
I see that around mid November 2013 a handful of my sites pages dropped off of Google completely. It was around the data refreshes in November, and while everyone says it doesn't effect that much I was wondering if anyone knew if it could knock some of my pages out of the rankings for a specific keyword. Note - we had previously held muliple listings for different pages on our site for this particular keyword. Google kept the highest ranking and knocked the lower ones off. See attached image of our keyword ranking history to see what I mean. DcJJM0M
Algorithm Updates | | franchisesolutions0 -
Did anyone else get a massive drop off in Bing recently?
Only for one set of my keywords. But I fell from first page to off the map in one fell swoop. Did this happen to anyone else? Should I be worried?
Algorithm Updates | | CleanEdisonInc0 -
Google site links on sub pages
Hi all Had a look for info on this one but couldn't find much. I know these days that if you have a decent domain good will often automatically put site links on for your home if someone searches for your company name, however has anyone seen these links appear for sub pages? For example, lets say I had a .com domain with /en /fr /de sub folders, each seoed for their location. If I were to then have domain.com/en/ as no1 in Google for my company in the UK would I be able to get site links under this or does it only work on the 'proper' homepage domain.com/ A client of mine wants to reorganise their website so they have different location sections ranking in different markets but they also want to keep having sitewide links as they like the look of it Thanks Carl
Algorithm Updates | | Grumpy_Carl0 -
Are you getting any action from Google +1 ?
If you have added google plus one to your website you can check on the impact by visiting your webmaster tools account. In your GWT account you will see a left menu item for "+1 Metrics". If you click on "Search Impact" you can see the CTR change attributed to +1. Anybody seeing anything there yet?
Algorithm Updates | | EGOL0 -
How long does a news article stay on Google's 'News' section on the SERP?
Our site is recognised as a news source for our niche - was just wondering if anyone had any idea how long the news story stays on the front page of the SERP once Google picks it up?
Algorithm Updates | | DanHill0