What to do about "blocked by meta-robots"?
-
The crawl report tells me "Notices are interesting facts about your pages we found while crawling". One of these interesting facts is that my blog archives are "blocked by meta robots".
Articles are not blocked, just the archives.
What is a "meta" robot?
I think its just normal (since the article need only be crawled once) but want a second opinion. Should I care about this?
-
Meta robots refers to the < meta name="robots" > tag at the page header level. This is usually the case when a blog is set up with an SEO program like All In One SEO for example, where you can manually set which content is blocked. It's common to block archives, tags, and other sections, in the theory that allowing these to be crawled could either cause duplicate content issues, or drain link value from the primary category navigation.
-
In general, there are two ways you can block crawlers from indexing your content.
-
You can add a Disallow entry to your robots.txt file
-
You can add a meta tag to your pages:
What you are saying in either case is "please do not list this content in your search engine".
In general, you would not want to block your archives. There certainly can be specific cases where you only want the public to see your most current content, in which case you can block it.
-
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Do Meta Keywords matter?
I am a firm believer in the fundamentals of SEO but is there any data to support its impact positively or negatively towards a sites rank?
Technical SEO | | Brandonp0 -
SEMRush's Site Audit Tool "SEO Ideas"
Recently SEMRush added a feature to its site audit tool called "SEO Ideas." In the case of specific the site I'm looking at it with, it's ideas consist mostly of suggesting words to add to the page for the page/my phrase(s) to perform better. It suggests this even when the term(s) or phrases(s) it's looking at are #1. Has anybody used this tool for this or something similar and found it to be valuable and if so how valuable? The reason I ask is that it would be a fair amount of work to go through these pages and find ways to add the select words and phrases and, frankly, it feels kind of 2005 to me. Your thoughts? Thanks... Darcy
Technical SEO | | 945010 -
Target="_blank"
Do href links that leave a site and use target="_blank" to open a new tab impact SEO?
Technical SEO | | ChristopherGlaeser0 -
What is the recommended or "best practice" Permalink Structure?
I have always been under the impression that by connecting pages to their parent pages as described in a.) below is best practice and makes sense to me. a.) yoursite.com/category/sub-category/product/ b.) yoursite.com/product But then i also understand the importance in terms of link juice being spread out across so many sub pages, and by using Example b.) you keep the link juice in tact. Your thoughts on this? Greg
Technical SEO | | AndreVanKets0 -
Rel="canonical" of .html/ to .html
Hi, could you guys confirm me that the following scenario is completely senseless? I just got the instruction from an external consultant (with quiet good SEO knowledge) to use a rel="canonical" for the following urls. http://www.example.com/petra.html/
Technical SEO | | petrakraft
to
http://www.example.com/petra.html I mean a folder petra/ to petra is ok - but a trailing slash after .html ??? Apart from that I would rather choose a 301 - not a rel canonical. What is your position here?0 -
Meta description and Meta Keywords
Hi, We are new to SEO and have some meta Q's Should Meta descriptions and meta keywords be different on every page? Is it bad to have the same meta data repeated on the site? If it has to be different does it have to be totally different per page of just slightly different? Should the description contain keywords is there an advantage to that? Thanks Andrew
Technical SEO | | Studio330 -
Google is Showing Website as "Untitled"
My freelance designer made some changes to my website and all of a sudden my homepage was showing the title I have in Dmoz. We thought maybe the NOODP tag was not correct, so we edited that a little and now the site is showing as "Untitled". The website is http://www.chemistrystore.com/. Of course he didn't save an old copy that we can revert to. That is a practice that will end. I have no idea why the title and description that we have set for the homepage is not showing in google when it previously was. Another weird thing that I noticed is that when I do ( site:chemistrystore.com ) in Google I get the https version of the site showing with the correct title and description. When I do ( site:www.chemistrystore.com ) in Google I don't have the hompage showing up from what I can tell, but there are 4,000+ pages to the site. My guess is that if it is showing up, it is showing up as "Untitled". My question is.... How can we get Google to start displaying the proper title and description again?
Technical SEO | | slangdon0