Meta robots
-
Hi,
I am checking a website for SEO and I've noticed that a lot of pages from the blog have the following meta robots:
meta name="robots" content="follow"
Normally these pages should be indexed, since search engines will index and follow by default. In this case however, a lot of pages from this blog are not indexed.
Is this because the meta robots is specified, but only contains follow? So will search engines only index and follow by default if there is no meta robots specified at all?
And secondly, if I would change the meta robots, should I just add index or remove the meta robots completely from the code?
Thanks for checking!
-
Thanks, this is a really helpful answer.
-
Hi Mat_C
There is no issue with that Meta Robots tag. This is not the reason why those pages aren't indexed.
I'd look a little deep trying to understand why Google didn't want to index that pages.
Do you have access to that website Search Console? What does index coverage report say?
Have you tried looking for one of those URLs in the "URL Inspection Tool"? There you might find why Google chose not to index it.That said, assuming that the site has as CMS Wordpress, the widely known YOAST plugin allows you to configure to be non-indexable many "known to cause issues" pages, such as tag or archive pages.
Have you checked that this is not the case?Also, there is another common reason why pages aren't indexed: Canonicals chosen by Google. This happens when some pages are almost identical and/or serve for the same user intent, so Google's Algorithms consider them as the same and just set one as the canonical for other, even when there isn't any canonical tag present.
Hope it helps.
Best luck.
GR -
I am pretty sure that's not how Meta robots tags work. If you fail to specify something, Google assumes they are allowed to index by default. By the way, search engines do not index pages which they don't think users will like or be interested in. Just because a search engine 'can' index a URL, that doesn't mean it will!
Follow directives and index directives actually operate on two entirely different sub-sets of data. Follow / nofollow directives are link-level (meaning they apply only to the hyperlinks on a page, not to the page itself). Index / no-index directives are page-level, and apply to the entire page upon which they are situated
Due to this, I don't believe they could or would interfere with each other in the way you described
Interesting experiment though. To test, I'd recommend adding index instead of removing follow. If hat doesn't make any kind of difference, it's not the issue
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Yoast platinum, news plug-in & meta properties
I have followed some of the discussions about Yoast and the news plug-in, but have not found specific information about the use of meta properties. One of our competitors is successfully using about 15 meta properties to gain news ranking. They list the publisher as Facebook. Is this coding part of the Yoast package or hard coding? As an example:
Intermediate & Advanced SEO | | jgodwin0 -
Wrong meta descriptions showing in the SERPS
We recently launched a new site on https, and I'm seeing a few errors in the SERPS with our meta descriptions as our pages are starting to get indexed. We have the correct meta data in our code but it's being output in Google differently. Example: http://imgur.com/ybqxmqg Is this just a glitch on Google's side or is there an obvious issue anyone sees that I'm missing? Thanks guys!
Intermediate & Advanced SEO | | Brian_Owens_10 -
Robots.txt Blocking - Best Practices
Hi All, We have a web provider who's not willing to remove the wildcard line of code blocking all agents from crawling our client's site (user-agent: *, Disallow: /). They have other lines allowing certain bots to crawl the site but we're wondering if they're missing out on organic traffic by having this main blocking line. It's also a pain because we're unable to set up Moz Pro, potentially because of this first line. We've researched and haven't found a ton of best practices regarding blocking all bots, then allowing certain ones. What do you think is a best practice for these files? Thanks! User-agent: * Disallow: / User-agent: Googlebot Disallow: Crawl-delay: 5 User-agent: Yahoo-slurp Disallow: User-agent: bingbot Disallow: User-agent: rogerbot Disallow: User-agent: * Crawl-delay: 5 Disallow: /new_vehicle_detail.asp Disallow: /new_vehicle_compare.asp Disallow: /news_article.asp Disallow: /new_model_detail_print.asp Disallow: /used_bikes/ Disallow: /default.asp?page=xCompareModels Disallow: /fiche_section_detail.asp
Intermediate & Advanced SEO | | ReunionMarketing0 -
What is better for Meta description ??
Hi everybody, I noticed that a lot of websites prefer their meta description would be the first words of the content inside.
Intermediate & Advanced SEO | | roeesa
I on the other hand thought that google will prefer the meta description to be like a peek to what going to be inside.
anyone can explain me, what is better? Thanks 🙂0 -
Weirdist Meta Description I've Seen in a SERP
For one e-commerce website, in place of the proper meta description, Google is showing a 318-character-long mix of snippets from the homepage content for the domain search (e.g. [example.com]). A brand search returns the correct meta description - as do the keywords the homepage ranks for. I know Google changes the meta description if it doesn't think it's relevant, but this one (there is only one) is and has (as far as we know) shown until now, and I've never seen such a mix of text in the SERP, and so many characters - it's picking up random text from bits of anchor text e.g. "privacy policy", title attributes from links, labels from radio buttons and more. The home page W3C validates apart from a couple of basic things like missing alt text. The only things that might be related that don't are some custom meta name tags added by the CMS - but I wouldn't think this would make any difference to whether a meta description is displayed properly or not? I've recommended we wait until tomorrow to see if Google fixes this on recrawl, but does anyone have any ideas if it doesn't? The homepage doesn't feature much standalone text, so I was thinking if we add a few extra words it might encourage Google to pick from that if it doesn't want to use the meta description. The text would have to be useful for users and fit in with the design of course, which could be awkward...
Intermediate & Advanced SEO | | Alex-Harford1 -
Whole site blocked by robots in webmaster tools
My URL is: www.wheretobuybeauty.com.auThis new site has been re-crawled over last 2 weeks, and in webmaster tools index status the following is displayed:Indexed 50,000 pagesblocked by robots 69,000Search query 'site:wheretobuybeauty.com.au' returns 55,000 pagesHowever, all pages in the site do appear to be blocked and over the 2 weeks, the google search query site traffic declined from significant to zero (proving this is in fact the case ).This is a Linux php site and has the following: 55,000 URLs in sitemap.xml submitted successfully to webmaster toolsrobots.txt file existed but did not have any entries to allow or disallow URLs - today I have removed robots.txt file completely URL re-direction within Linux .htaccess file - there are many rows within this complex set of re-directions. Developer has double checked this file and found that it is valid.I have read everything that google and other sources have on this topic and this does not help. Also checked webmaster crawl errors, crawl stats, malware and there is no problem there related to this issue.Is this a duplicate content issue - this is a price comparison site where approx half the products have duplicate product descriptions - duplicated because they are obtained from the suppliers through an XML data file. The suppliers have the descriptions from the files in their own sites.Help!!
Intermediate & Advanced SEO | | rrogers0 -
How to Disallow Tag Pages With Robot.txt
Hi i have a site which i'm dealing with that has tag pages for instant - http://www.domain.com/news/?tag=choice How can i exclude these tag pages (about 20+ being crawled and indexed by the search engines with robot.txt Also sometimes they're created dynamically so i want something which automatically excludes tage pages from being crawled and indexed. Any suggestions? Cheers, Mark
Intermediate & Advanced SEO | | monster990 -
Robots.txt 404 problem
I've just set up a wordpress site with a hosting company who only allow you to install your wordpress site in http://www.myurl.com/folder as opposed to the root folder. I now have the problem that the robots.txt file only works in http://www.myurl./com/folder/robots.txt Of course google is looking for it at http://www.myurl.com/robots.txt and returning a 404 error. How can I get around this? Is there a way to tell google in webmaster tools to use a different path to locate it? I'm stumped?
Intermediate & Advanced SEO | | SamCUK0