Warnings for blocked by blocked by meta-robots/meta robots Nofollow...how to resolve?
-
Hello,
I see hundreds of notices for blocked by meta-robots/meta robots nofollow and it appears it is linked to the comments on my site which I assume I would not want to be crawled. Is this the case and these notices are actually a positive thing? Please advise how to clear them up if these notices can be potentially harmful for my SEO.
Thanks,
Talia
-
The notices in the SEOmoz PRO platform are just that - notices. They are neither good nor bad in themselves, but we want you to be aware of them in case the issues are not supposed to be there.
In the case of comments such as yours, there are very legitimate reasons why you wouldn't want these crawled or indexed, depending on your situation (duplicate content issues, link to spam, etc) If you are aware why the issues are appearing, and comfortable with your reasoning, it's perfectly fine to have notices in your account.
-
If you are talking about Google Webmaster reporting that blog comments are blocked or no-followed, then yes, that is a good thing. Normally, it's a pretty good idea to make sure that any links that are added to blog comments are no-follow.
However, you do want to make sure that you are not blocking the blog post itself from being indexed or crawled. Just double-check the URL that Webmaster is reporting. If it's just for comment links, then your all good. If it's for the whole URL, then you might want to consider modifying your robots.txt file to make sure that your blog page is crawled but comment links are not.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Nofollow/Noindex Category Listing Pages with Filters
Our e-commerce site currently has thousands of duplicate pages indexed because category listing pages with all the different filters selected are indexed. So, for example, you would see indexed: example.com/boots example.com/boots/black example.com/boots/black-size-small etc. There is a logic in place that when more than one filter is selected all the links on the page are nofollowed, but Googlebot is still getting to them, and the variations are being indexed. At this point I'd like to add 'noindex' or canonical tags to the filtered versions of the category pages, but many of these filtered pages are driving traffic. Any suggestions? Thanks!
Technical SEO | | fayfr0 -
Robots.txt & meta noindex--site still shows up on Google Search
I have set up my robots.txt like this: User-agent: *
Technical SEO | | RoxBrock
Disallow: / and I have this meta tag in my on a Wordpress site, set up with SEO Yoast name="robots" content="noindex,follow"/> I did "Fetch as Google" on my Google Search Console My website is still showing up in the search results and it says this: "A description for this result is not available because of this site's robots.txt" This site has not shown up for years and now it is ranking above my site that I want to rank for this keyword. How do I get Google to ignore this site? This seems really weird and I'm confused how a site with little content, that has not been updated for years can rank higher than a site that is constantly updated and improved.1 -
Hey all -- ever seen a client with URLs that keep repeating the domain? Something like: client.com/client.com/client.com/subfolder-name. Any idea what glitch could cause that?
Hey all -- ever seen a client with URLs that keep repeating the domain? Something like: client.com/client.com/client.com/subfolder-name. Any idea what glitch could cause that?
Technical SEO | | TDC_SEO0 -
No descripton on Google/Yahoo/Bing, updated robots.txt - what is the turnaround time or next step for visible results?
Hello, New to the MOZ community and thrilled to be learning alongside all of you! One of our clients' sites is currently showing a 'blocked' meta description due to an old robots.txt file (eg: A description for this result is not available because of this site's robots.txt) We have updated the site's robots.txt to allow all bots. The meta tag has also been updated in WordPress (via the SEO Yoast plugin) See image here of Google listing and site URL: http://imgur.com/46wajJw I have also ensured that the most recent robots.txt has been submitted via Google Webmaster Tools. When can we expect these results to update? Is there a step I may have overlooked? Thank you,
Technical SEO | | adamhdrb
Adam 46wajJw0 -
Very strange: META descriptions not showing
Hello, Since Panda 4.0 has been launched, all of my optimized META description have been gone in Google.
Technical SEO | | MarcelMoz
A while ago, I posted a question about this problem here: http://moz.com/community/q/all-meta-descriptions-gone. I know about Google's own will to decide which META description will be shown. And also about unique content of the descriptions. All pages did have an optimized description before Panda 4.0 and there were no troubles at all, what tells me there is something else going on. I tested some things: Rewrote 50 descriptions to very uinique ones, only five got indexed. This tells me that duplicate content of the descriptions is not the problem (they have never been 100% duplicate, product type was a variable which was always different for each page). Removed cache in GWT and fetched again as Google, didn't help. I checked the pages I tested and they all have been indexed again without showing the optimized descriptions. More information: The first time I changed some META descriptions and fetched the pages again in GWT, Google picked up my new META descriptions and showed them. A few days later, most of them disappeared again (so Google is aware of the description but seems to ignore it). Some pages show the optimized description when I change my search query (only a few, mostly the optimized description never got shown) Technique is ok. Source code shows the right optimized description. META robots isn't blocking anything except NOODP/NOYDIR (always has blocked those). Websites using the exact same CMS, website template, META descriptions (style and build-up), do not have these problems I compared elements like place of description in source code, usage of meta robots, og:description, crawl-delay in robots.txt, and special characters in descriptions between websites that are showing optimized vs. website that don't show optimized descriptions. I can't find any connection. Something I noticed is a change is my Robots.txt file: my webmaster has added the following command:
Crawl-delay: 2 May this have to do with my problem? I guess it doens't. I did some research and there are more websites that are suffering this problem beside mine. This tells me it must be Google (and so Panda 4.0) that is responsible for this change. I realy want my optimized descriptions back. Does anybody have an idea what to do?
Thanks in advance. Marcel0 -
Sub Domains and Robot.txt files...
This is going to seem like a stupid question, and perhaps it is but I am pulling out what little hair I have left. I have a sub level domain on which a website sits. The Main domain has a robots.txt file that disallows all robots. It has been two weeks, I submitted the sitemap through webmaster tools and still, Google has not indexed the sub domain website. My question is, could the robots.txt file on the main domain be affecting the crawlability of the website on the sub domain? I wouldn't have thought so but I can find nothing else. Thanks in advance.
Technical SEO | | Vizergy0 -
Rebranding / Redirecting
Hi I have a client who wants to re-brand their shopify clothing store under new domain name. Whilst still a clothing store its going to have different department structure and product pages and will hence be a different store/site. Is there anyway to pass any of the history/authority of existing site to the new one such as 301 redirecting the top level pages of existing site to nearest equivalent pages of new site etc or best to just redirect the old site domain to the new ? Cheers Dan
Technical SEO | | Dan-Lawrence0 -
Mobile site - allow robot traffic
Hi, If a user comes to our site from a mobile device, we redirect to our mobile site. That is www.mysite/mypage redirects to m.mysite/mypage. Right now we are blocking robots from crawling our m. site. Previously there were concerns the m. site could rank for normal browser searches. To make sure this isn't a problem we are planning on rel canonical our m. site pages and reference the www pages (mobile is just a different version of our www site). From my understanding having a mobile version of a page is a ranking factor for mobile searches so allowing robots is a good thing. Before doing so, I wanted to see if anyone had any other suggestions/feedback (looking for potential pitfalls, issues etc)
Technical SEO | | NicB10