Meta robots at every page rather than using robots.txt for blocking crawlers? How they'll get indexed if we block crawlers?
-
Hi all,
The suggestion to use meta robots tag rather than robots.txt file is to make sure the pages do not get indexed if their hyperlinks are available anywhere on the internet. I don't understand how the pages will be indexed if the entire site is blocked? Even though there are page links are available, will Google really index those pages? One of our site got blocked from robots file but internal links are available on internet for years which are not been indexed. So technically robots.txt file is quite enough right? Please clarify and guide me if I'm wrong.
Thanks
-
I agree with Gaston's approach right up to step 4. If you add the no-indexed pages back into a block in the robots.txt file, you'll end up back where you started from. Because Google will still discover the no-indexed URLs elsewhere and the robots,txt block will stop them from discovering the no-index, and the URLs will likely start to get added to the index again.
No-indexed URLs must not be blocked in robots.txt. Those two processes are mutually exclusive.
-
Hi there,
TLDR; The solution to deindexing and never index again:
- Allow (with robots.txt) the web to be crawable
- Aplly meta robots tag: noindex,follow
- Wait somte weeks to be completely deindexed
- block the entire site/section with robots.txt
Robots.txt and the robots meta tag can make the same effect, but to understand them must be analyzed separatedly.
-
Robots.txt, here you just tell bots where they can go BEFORE they crawl any of the website. This is just a signal, not a directive... Because robots can choose to ignore the what's in the file. Here you can block from the entire web, to an entire section or just specific pages. More info: Robots.txt official page and a really cool and complete guide to robots.txt
-
Robots meta tag, with it you have more signals to tell, the most used are: noindex, nofollow and follow, due to the usual issues about indexing. More info: Robots.txt offical page, Google developers, Meta Robots directive - Moz and a complete guide to meta robots tag - YOAST.
Hope this is what you wanted.
Best luck
GR.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Do orphan pages take away link juice?
Hi, Just wondering about this whether the orphan pages take away any link juice? We been creating lot of them these days only to link from external sites as landing pages on our site. So, not linking from any part of our website; just linking from other websites. Also, will they get any link juice if they are linked from our own blog-post? Thanks
Algorithm Updates | | vtmoz1 -
Does personalization that changes meta data display in SERPs impact SEO?
My company has been rolling out personalization at the page level across our site using behavior paths embedding content from cross pathed pages as well as customer journey mapping. The dynamically generated content doesn’t change the URLs. In the SERPs I’m seeing that our title tags and meta descriptions also seem to be dynamically generated even though we have these elements crafted. The way our elements are crafted: Title tag: descriptive Keyword rich phrase | Brand Meta description: Keyword rich, grammatically correct description tied to title tag and page content for consistency. I search a specific URL: Title tag display: Keyword rich phrase | Brand – Brand Meta description display: Random content pulled from the page I search a phrase that includes Brand + keywords in the URL: Title tag display: Title tag we crafted Meta description display: Meta description we crafted I search a phrase that includes Brand + keywords in the title tag: Title tag display: Title tag we crafted Meta description display: Random content pulled from the page Does Google crawl the page and digest the title tag and meta description we crafted? Or is Google going to ding us for having the brand twice, exceeding the length of the title tag, etc.? I have been searching the interwebs, forums and the cosmos, but the only information I’m finding is related to the fact that URLs are changing and how that would impact SEO. That’s not the case for us. Thoughts on how all this is impacting our SEO efforts?
Algorithm Updates | | NStarJM0 -
URL in SERP: Google's stand
Months back, we can notice "keyword" will be bold and highlighted if its in the SERP URL. Now Google no more highlights any URLs even with exact match of keyword we search. Beside UI, Does this mean Google might devalued or reduced the importance of URL as ranking factor? We can see many search results match partially or completely in URL with search keywords.
Algorithm Updates | | vtmoz0 -
Organic listing & map listing on 1st page of Google
Hi, Back then, a company could get multiple listings in SERP, one in Google Maps area and a homepage or internal pages from organic search results. But lately, I've noticed that Google are now putting together the maps & organic listings. This observation has been confirmed by a couple of SEO people and I thought it made sense, but one day I stumble with this KWP "bmw dealership phoenix" and saw that www.bmwnorthscottsdale.com has separate listing for google places and organic results. Any idea how this company did this? Please see the attached image
Algorithm Updates | | ao5000000 -
Is it OK to 301 redirect the index page to a search engine friendly url
Is it OK to 301 redirect the index page to a search engine friendly url.
Algorithm Updates | | WinningInch0 -
Infographics Links could get discounted in the future
Hey guys, I read this article this morning on SEL. Not sure what to think about it.. Matt did have a point that a lot of infographics are of bad quality (even with wrong information present at times) , and hence don't deserve to gain links from it. But how could Google possible know whether the infographic itself is of high quality or not?? http://searchengineland.com/cutts-infographic-links-might-get-discounted-in-the-future-127192
Algorithm Updates | | Michael-Goode0 -
"Revisit-after" Metatag = Why use it?
Hi Mozfans, Just been thinking about the robots revisit metatag, all pages on my website (200+ pages) have the following tag on them; name="revisit-after" content="7 days" /> I'm wondering what is the purpose of the tag? Surely isn't it best to allow robots (such as Googlebot or Bingbot) to crawl your site as often as possible so the index and rankings get updated as quickly as possible? Thanks in advance everyone! Ash
Algorithm Updates | | AshSEO20110 -
No-follow tags on links in the footer...do it or don't do it?
With some of the great reports SEOMoz has provided I've been able to start to take the correct steps towards fixing crawl issues, on-page issues, etc. One of my websites allows a customer to drill down to their specific state and then their city to apply for an auto loan. The SEOMoz reports told me I had too many links on these pages specifically. One of my ways to remedy this would be to add "no-follow" tags on the links in the footer as well as the links to the cities. Am I steering myself in the right/wrong direction? Should I be approaching this problem from a different perspective? Any help is greatly appreciated!
Algorithm Updates | | fergseo0