Impact of "restricted by robots" crawler error in WT
-
I have been wondering about this for a while now with regards to several of my sites. I am getting a list of pages that I have blocked in the robots.txt file. If I restrict Google from crawling them, then how can they consider their existence an error? In one case, I have even removed the urls from the index.
And do you have any idea of the negative impact associated with these errors.
And how do you suggest I remedy the situation.
Thanks for the help
-
Google is just showing you a warning that hey, these are excluded, make sure that you want them excluded. They're not passing a judgement on whether or not they should be excluded. So, as long as they're excluded on purposes, no worries.
-
Hi Patrick,
That section is simply there to advice on any URLs that Google feels are wrongly excluded within the robots.txt
If the URLs are not wrongly excluded, don't worry about it showing in WMT's - it's there just as an advisory.
Good luck!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Menu impact on SEO
we have a single page web application for an ecommerce website. I think it is built in angular. One UX features we are exploring is the use of a "Products" Item on the menu with the categories showing on a menu rather than directly present on the header. The aim being to keep the header nice and clean. The result of this is that the categories which would typically sit in the header will now not be immediately visible until the menu is opened. Let's say I want to rank well for "building materials". Traditionally the view would be that this word would need to be in the header and marked up with the appropriate h tag. Will moving "building materials" into a product menu be detrimental for SEO? My initial thought is that as long as it is coded correctly there shouldn't be any impact on SEO. Can anyone give me their expert SEO view?
Technical SEO | | built_bot0 -
Robots and Canonicals on Moz
We noticed that Moz does not use a robots "index" or "follow" tags on the entire site, is this best practice? Also, for pagination we noticed that the rel = next/prev is not on the actual "button" rather in the header Is this best practice? Does it make a difference if it's added to the header rather than the actual next/previous buttons within the body?
Technical SEO | | PMPLawMarketing0 -
Meta-robots Nofollow
I don't understand Meta-robots Nofollow. Wordpress has my homepage set to this according to SEOMoz tool. Is this really bad?
Technical SEO | | hopkinspat1 -
I was googling the word "best web hosting" and i notice the 1st and 3rd result were results with google plus. Does Google plus now play a role in improving ranking for the website?
I was googling the word "best web hosting" and i notice the 1st and 3rd result were results with google plus. Does Google plus now play a role in improving ranking for the website?I see a person's name next to the website too
Technical SEO | | mainguy0 -
Cn I use SEOMOZ to find "Bad Links"
We were hit by the Penguin update and I am told it make be because of "Bad Links", but no one can seem to tell me how to find them. We never buy links, and in fact the only links I know about are those from paid affiliates through shareasale - and these affiliates are paid based on performance, not links. 1. Does anyone know how to figure out what links are bad? 2. Once I know, how do I get them to stop linking to my site? Thanks!
Technical SEO | | trophycentraltrophiesandawards0 -
Crawler Stats
Hello, On all of my crawler stats, it is showing both http://domainhere.com and http://www.domainhere.com Is it bad to have both types of URLS (one with www and one without www) or should we be only using links with www? This question may not make any sense, but pretty much should it matter that both versions are showing up in my crawler results or not? Thanks for any help in advance.
Technical SEO | | EQ-Richie0 -
International Websites: rel="alternate" hreflang="x"
Hi people, I keep on reading and reading , but I won't get it... 😉 I mean this page: http://support.google.com/webmasters/bin/answer.py?hl=en&answer=189077&topic=2370587&ctx=topic On the bottom of the page they say: Step 2: Use rel="alternate" hreflang="x" Update the HTML of each URL in the set by adding a set of rel="alternate" hreflang="x" link elements. Include a rel="alternate" hreflang="x" link for every URL in the set, like this: This markup tells Google's algorithm to consider all of these pages as alternate versions of each other. OK! Each URL needs this markup. BUT: Do i need it exactly as written above, or do I have to put in the complete URL of the site, like: The next question is, what happens exactly in the SERPS when I do it like this (an also with Step1 that I haven't copied here)? Google will display the "canonical"-version of the page, but wehen a user from US clicks he will get on http://en-us.example.com/**page.htm **??? I tried to find other sites which use this method, but I haven't found one. Can someone give me an example.website??? Thank you, thank you very much! André
Technical SEO | | waynestock0 -
Does Google follow links in "id" tag?
Hello, For functionality purposes, I need to wrap separate blocks of content with a tag. The main question is whether Google will follow this URL, even though it is not a hyperlink on the page, just a URL used for functionality purposes. We will have 10-20 of these types of span tags with a different URL for each one, and we just want to be sure that Google will not be following these URLs that are not links. Thanks!
Technical SEO | | Hakkasan0