Meta robots
-
Hi,
I am checking a website for SEO and I've noticed that a lot of pages from the blog have the following meta robots:
meta name="robots" content="follow"
Normally these pages should be indexed, since search engines will index and follow by default. In this case however, a lot of pages from this blog are not indexed.
Is this because the meta robots is specified, but only contains follow? So will search engines only index and follow by default if there is no meta robots specified at all?
And secondly, if I would change the meta robots, should I just add index or remove the meta robots completely from the code?
Thanks for checking!
-
Thanks, this is a really helpful answer.
-
Hi Mat_C
There is no issue with that Meta Robots tag. This is not the reason why those pages aren't indexed.
I'd look a little deep trying to understand why Google didn't want to index that pages.
Do you have access to that website Search Console? What does index coverage report say?
Have you tried looking for one of those URLs in the "URL Inspection Tool"? There you might find why Google chose not to index it.That said, assuming that the site has as CMS Wordpress, the widely known YOAST plugin allows you to configure to be non-indexable many "known to cause issues" pages, such as tag or archive pages.
Have you checked that this is not the case?Also, there is another common reason why pages aren't indexed: Canonicals chosen by Google. This happens when some pages are almost identical and/or serve for the same user intent, so Google's Algorithms consider them as the same and just set one as the canonical for other, even when there isn't any canonical tag present.
Hope it helps.
Best luck.
GR -
I am pretty sure that's not how Meta robots tags work. If you fail to specify something, Google assumes they are allowed to index by default. By the way, search engines do not index pages which they don't think users will like or be interested in. Just because a search engine 'can' index a URL, that doesn't mean it will!
Follow directives and index directives actually operate on two entirely different sub-sets of data. Follow / nofollow directives are link-level (meaning they apply only to the hyperlinks on a page, not to the page itself). Index / no-index directives are page-level, and apply to the entire page upon which they are situated
Due to this, I don't believe they could or would interfere with each other in the way you described
Interesting experiment though. To test, I'd recommend adding index instead of removing follow. If hat doesn't make any kind of difference, it's not the issue
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Tool to identify if meta description are showing?
Hi we have a Ecommerce client with 1000s of meta descriptions, we have noticed that some meta descriptions are not showing properly, we want to pull and see which ones are showing on Google SERP results. You can use tools like screaming frog to pull meta description from page, but we want to see if it's showing for certain keywords. Any ideas on how to automate this? Cheers.
Intermediate & Advanced SEO | | brianna00 -
Not sure how we're blocking homepage in robots.txt; meta description not shown
Hi folks! We had a question come in from a client who needs assistance with their robots.txt file. Metadata for their homepage and select other pages isn't appearing in SERPs. Instead they get the usual message "A description for this result is not available because of this site's robots.txt – learn more". At first glance, we're not seeing the homepage or these other pages as being blocked by their robots.txt file: http://www.t2tea.com/robots.txt. Does anyone see what we can't? Any thoughts are massively appreciated! P.S. They used wildcards to ensure the rules were applied for all locale subdirectories, e.g. /en/au/, /en/us/, etc.
Intermediate & Advanced SEO | | SearchDeploy0 -
Need help with Robots.txt
An eCommerce site built with Modx CMS. I found lots of auto generated duplicate page issue on that site. Now I need to disallow some pages from that category. Here is the actual product page url looks like
Intermediate & Advanced SEO | | Nahid
product_listing.php?cat=6857 And here is the auto generated url structure
product_listing.php?cat=6857&cPath=dropship&size=19 Can any one suggest how to disallow this specific category through robots.txt. I am not so familiar with Modx and this kind of link structure. Your help will be appreciated. Thanks1 -
Should I be using meta robots tags on thank you pages with little content?
I'm working on a website with hundreds of thank you pages, does it make sense to no follow, no index these pages since there's little content on them? I'm thinking this should save me some crawl budget overall but is there any risk in cutting out the internal links found on the thank you pages? (These are only standard site-wide footer and navigation links.) Thanks!
Intermediate & Advanced SEO | | GSO0 -
meta robots no follow on page for paid links
Hi I have a page containing paid links. i would like to add no follow attribute to these links
Intermediate & Advanced SEO | | Kung_fu_Panda
but from technical reasons, i can only place meta robots no follow on page level (
is that enough for telling Google that the links in this page are paid and and to prevent Google penlizling the sites that the page link to? Thanks!0 -
Robots Disallow Backslash - Is it right command
Bit skeptical, as due to dynamic url and some other linkage issue, google has crawled url with backslash and asterisk character ex - www.xyz.com/\/index.php?option=com_product www.xyz.com/\"/index.php?option=com_product Now %5c is the encoded version of \ - backslash & %22 is encoded version of asterisk Need to know for command :- User-agent: * Disallow: \As am disallowing all backslash url through this - will it only remove the backslash url which are duplicates or the entire site,
Intermediate & Advanced SEO | | Modi0 -
Do different meta titles & descriptions delete the canonical origin?
Hi, hopefully anyone knows something about this case: There is a canonical tag on site "www.xyz.com**/de_de/" **refering to site "www.xyz.com/de-de/". If the meta title and descriptions are different on both sides - is there a problem that google will not pay attention to the canonical tag? Do both sides need the same title and canonical? Thanx for your answers! Cheers Heiko!
Intermediate & Advanced SEO | | heckert0 -
Using 2 wildcards in the robots.txt file
I have a URL string which I don't want to be indexed. it includes the characters _Q1 ni the middle of the string. So in the robots.txt can I use 2 wildcards in the string to take out all of the URLs with that in it? So something like /_Q1. Will that pickup and block every URL with those characters in the string? Also, this is not directly of the root, but in a secondary directory, so .com/.../_Q1. So do I have to format the robots.txt as //_Q1* as it will be in the second folder or just using /_Q1 will pickup everything no matter what folder it is on? Thanks.
Intermediate & Advanced SEO | | seo1234560