Meta-robots Nofollow on logins and admins
-
In my SEO MOZ reports I am getting over 400 errors as Meta-robots Nofollow. These are all leading to my admin login page which I do not want robots in. Should I put some code on these pages so the robots know this and don't attempt to and I do not get these errors in my reports?
-
I think that's probably fine - we're warning you just to tell you the pages are blocked, but blocking them sounds perfectly sensible. Is Google indexing them at all?
I assume that the login page is getting multiple URLs, possibly based on the path people take to it? Ideally, it would help to collapse that to one URL (store the parameters in a form field or visitor session/cookie), but my gut reaction is that that's a nice to have. I suspect you're probably ok.
-
Yes you can noindex that page to prevent robots from entering. Go to your robot.txt file and add "disallow with your URL. http://www.robotstxt.org/robotstxt.html
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google is not respecting the meta title
We're experiencing a peculiar situation with Google not respecting our meta <title>.</p> <p>As you can see in the first image (search result), the title <a href="http://open.iebschool.com/profesores/startups/">for the page</a> is a part of the content. This is relatevely normal for the description, but we never heard of Google doing this before.</p> <p>In the code, the <title> and meta description are correctly implemented.</p> <blockquote style="background-color: #f7f7f7; padding-top: 5px; margin-left: 0px; padding-left: 2px; padding-bottom: 5px; white-space: nowrap; overflow-y: auto; font-family: monospace; background-position: initial initial; background-repeat: initial initial;"> <p><meta name="description" content="Profesores, tutores, autores y docentes 2.0 de Open IEBS. Conoce su Biografía, experiencia, reputación, conexiones sociales y las valoraciones de alumnos."/><br /><title>Conoce los profesores, tutores, autores y docentes de Open IEBS.</title> In a further research, we discovered that the title which is using is an in anwith the following code (cleaned and simplified for the question): <hgroup> Pilar Soro
Technical SEO | | ofuente
0 Seguidor
Para poder seguir al Profesor, debes de registrarte aquí. Profesora y experta en redes sociales. Formadora de docentes, [...]
</hgroup> Note: we're correcting the code since this is quite messy, but it's the one we have now The point is that google has considered that this particular is more important than the title itself. This would make sense if we were looking for that name, but the search was simply "site:domain.com". Two things for which this is even more strange are the following: while all the /profesor/%category%/ has the same code, this only happens in some search results and not in all of them; why is it appearing in some pages, but respecting my title in others? the previous code is not the only one in the page, there are about 10 others and some are placed before and some are placed after; so, why this one and not the first or the last? What is more strange is why this article in particular and not any other of the 10 on the page since some of them are placed before and some of them are placed after. Provided this situation, we would like to know: is this a common situation? Is it happening to more people? why is it happening? Is it somehow related to , <hgroup>and ? why that piece of code and not any other article? and why is it only happening in some pages? more important, can it be corrected or can we take advantage of it somehow? Thank you in advance. Any light you can shed on this will be well received! AJ2CUSe.png?1?8232 </hgroup>0 -
Blocking Affiliate Links via robots.txt
Hi, I work with a client who has a large affiliate network pointing to their domain which is a large part of their inbound marketing strategy. All of these links point to a subdomain of affiliates.example.com, which then redirects the links through a 301 redirect to the relevant target page for the link. These links have been showing up in Webmaster Tools as top linking domains and also in the latest downloaded links reports. To follow guidelines and ensure that these links aren't counted by Google for either positive or negative impact on the site, we have added a block on the robots.txt of the affiliates.example.com subdomain, blocking search engines from crawling the full subddomain. The robots.txt file is the following code: User-agent: * Disallow: / We have authenticated the subdomain with Google Webmaster Tools and made certain that Google can reach and read the robots.txt file. We know they are being blocked from reading the affiliates subdomain. However, we added this affiliates subdomain block a few weeks ago to the robots.txt, but links are still showing up in the latest downloads report as first being discovered after we added the block. It's been a few weeks already, and we want to make sure that the block was implemented properly and that these links aren't being used to negatively impact the site. Any suggestions or clarification would be helpful - if the subdomain is being blocked for the search engines, why are the search engines following the links and reporting them in the www.example.com subdomain GWMT account as latest links. And if the block is implemented properly, will the total number of links pointing to our site as reported in the links to your site section be reduced, or does this not have an impact on that figure?From a development standpoint, it's a much easier fix for us to adjust the robots.txt file than to change the affiliate linking connection from a 301 to a 302, which is why we decided to go with this option.Any help you can offer will be greatly appreciated.Thanks,Mark
Technical SEO | | Mark_Ginsberg0 -
Is there any value in having a blank robots.txt file?
I've read an audit where the writer recommended creating and uploading a blank robots.txt file, there was no current file in place. Is there any merit in having a blank robots.txt file? What is the minimum you would include in a basic robots.txt file?
Technical SEO | | NicDale0 -
Should I block robots from URLs containing query strings?
I'm about to block off all URLs that have a query string using robots.txt. They're mostly URLs with coremetrics tags and other referrer info. I figured that search engines don't need to see these as they're always better off with the original URL. Might there be any downside to this that I need to consider? Appreciate your help / experiences on this one. Thanks Jenni
Technical SEO | | ShearingsGroup0 -
Duplicate Meta Descriptions From Pages That Don't Exist
Hi Guys I am hoping someone can help me out here. I have had a new site built with a unique theme and using wordpress as the CMS. Everything was going fine but after checking webmaster tools today I noticed something that I just cannot get my head around. Basically I am getting warnings of Duplicate page warnings on a couple of things. 1 of which i think i can understand but do not know how to get the warning to go. Firstly I get this warning of duplicate meta desciption url 1: / url 2: /about/who-we-are I understand this as the who-we-are page is set as the homepage through the wordpress reading settings. But is there a way to make the dup meta description warning disappear The second one I am getting is the following: /services/57/ /services/ Both urls lead to the same place although I have never created the services/57/ page the services/57/ page does not show on the xml sitemap but Google obviously see it because it is a warning in webmaster tools. If I press edit on services/57/ page it just goes to edit the /services/ page/ is there a way I can remove the /57/ page safely or a method to ensure Google at least does not see this. Probably a silly question but I cannot find a real comprehensive answer to sorting this. Thanks in advance
Technical SEO | | southcoasthost0 -
Long Meta Descriptions
I want to create a template for Meta titles, descriptions and keywords on my website for old news and minor pages in order to get some long tail traffic from them. The only template I can think to use for the descriptions takes the first sentence of the news article (which often if above 160 characters). Since these are minor pages, how big of a problem is that? Thanks!
Technical SEO | | theLotter0 -
Site not being Indexed that fast anymore, Is something wrong with this Robots.txt
My wordpress site's robots.txt used to be this: User-agent: * Disallow: Sitemap: http://www.domainame.com/sitemap.xml.gz I also have all in one SEO installed and other than posts, tags are also index,follow on my site. My new posts used to appear on google in seconds after publishing. I changed the robots.txt to following and now post indexing takes hours. Is there something wrong with this robots.txt? User-agent: * Disallow: /cgi-bin Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content/plugins Disallow: /wp-content/cache Disallow: /wp-content/themes Disallow: /wp-login.php Disallow: /wp-login.php Disallow: /trackback Disallow: /feed Disallow: /comments Disallow: /author Disallow: /category Disallow: */trackback Disallow: */feed Disallow: */comments Disallow: /login/ Disallow: /wget/ Disallow: /httpd/ Disallow: /*.php$ Disallow: /? Disallow: /*.js$ Disallow: /*.inc$ Disallow: /*.css$ Disallow: /*.gz$ Disallow: /*.wmv$ Disallow: /*.cgi$ Disallow: /*.xhtml$ Disallow: /? Disallow: /*?Allow: /wp-content/uploads User-agent: TechnoratiBot/8.1 Disallow: ia_archiverUser-agent: ia_archiver Disallow: / disable duggmirror User-agent: duggmirror Disallow: / allow google image bot to search all imagesUser-agent: Googlebot-Image Disallow: /wp-includes/ Allow: /* # allow adsense bot on entire siteUser-agent: Mediapartners-Google* Disallow: Allow: /* Sitemap: http://www.domainname.com/sitemap.xml.gz
Technical SEO | | ideas1230 -
Subdomain Robots.txt
If I have a subdomain (a blog) that is having tags and categories indexed when they should not be, because they are creating duplicate content. Can I block them using a robots.txt file? Can I/do I need to have a separate robots file for my subdomain? If so, how would I format it? Do I need to specify that it is a subdomain robots file, or will the search engines automatically pick this up? Thanks!
Technical SEO | | JohnECF0