Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

Meta Robots Noindex and Robots.txt File

Technical SEO

473

iskq last edited by

Can we use meta robots noindex and robots.txt together?
1 Reply Last reply
Reply Quote 0
Devanur-Rafi last edited by

Hi, let us take a scenario. test.html has been blocked using robots.txt file. In this scenario, the bots that respect the robots exclusion protocol will not crawl the page to encounter the page level robots noindex attribute. So does not make any sense to use both the page level robots exclusion method and server side robots.txt file.

Here you go for more:

http://support.google.com/webmasters/bin/answer.py?hl=en&answer=93708

And here:

http://support.google.com/webmasters/bin/answer.py?hl=en&answer=93710

Best regards,

Devanur Rafi.
1 Reply Last reply
Reply Quote 2

Got a burning SEO question?

Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.

Start my free trial

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

Blog archive pages are meta noindexed but still flagged as duplicate

Hi all. I know there several threads related to noindexing blog archives and category pages, so if this has already been answered, please direct me to that post. My blog archive pages have preview text from the posts. Each time I post a blog, the last post on any given archive page shifts to the first spot on the next archive page. Moz seems to report these as new duplicate content issues each week. I have my archive pages set to meta noindex, so can I feel good about continuing to ignore these duplicate content issues, or is there something else I should be doing to prevent penalties? TIA!
Technical SEO | | mkupfer

1
How to stop robots.txt restricting access to sitemap?

I'm working on a site right now and having an issue with the robots.txt file restricting access to the sitemap - with no web dev to help, I'm wondering how I can fix the issue myself? The robots.txt page shows User-agent: * Disallow: / And then sitemap: with the correct sitemap link
Technical SEO | | Ad-Rank

0
One robots.txt file for multiple sites?

I have 2 sites hosted with Blue Host and was told to put the robots.txt in the root folder and just use the one robots.txt for both sites. Is this right? It seems wrong. I want to block certain things on one site. Thanks for the help, Rena
Technical SEO | | renalynd27

0
Does adding a noindex tag reduce duplicate content?

I've been working under the assumption for some time that if I have two (or more) pages which are very similar that I can add a noindex tag to the pages I don't need and that will reduce duplicate content. As far as I know this removes the pages with the tag from Google's index and stops any potential issues with duplicate content. It's the second part of that assumption that i'm now questioning. Despite pages having the noindex tag they continue to appear in Google Search console as duplicate content, soft 404 etc. That is, new pages are appearing regularly that I know to have the noindex tag. My thoughts on this so far are that Google can still crawl these pages (although won't index them) so shows them in GSC due to a crude issue flagging process. I mainly want to know: a) Is the actual Google algorithm sophisticated enough to ignore these pages even through GSC doesn't. b) How do I explain this to a client.
Technical SEO | | ChrisJFoster

0
Is there any value in having a blank robots.txt file?

I've read an audit where the writer recommended creating and uploading a blank robots.txt file, there was no current file in place. Is there any merit in having a blank robots.txt file? What is the minimum you would include in a basic robots.txt file?
Technical SEO | | NicDale

0
Should I block robots from URLs containing query strings?

I'm about to block off all URLs that have a query string using robots.txt. They're mostly URLs with coremetrics tags and other referrer info. I figured that search engines don't need to see these as they're always better off with the original URL. Might there be any downside to this that I need to consider? Appreciate your help / experiences on this one. Thanks Jenni
Technical SEO | | ShearingsGroup

0
Should I noindex, follow categories?

Hey Everyone, A simple question (hopefully). Should I check or uncheck the noindex, follow setting for categories on our site? We've got about 5-6 but they aren't anything that people should know or would help in SEO. For example, one category is "featured content" and another is "what's happening." Checking them dictate where a post goes on the site. I'm pretty sure it should be checked, but I wanted to check with the experts first 🙂 http://d.pr/i/jtrc
Technical SEO | | ttb

0
Meta tags - better NOT to have?

OK ok . . . the SEOMox report card told me it's actually better NOT to have meta tag keywords on my page, because my competitors can then look at my page to see what words I am trying to target . . . That makes since, but is also painfully counter intuitive. I thought I would just double check and make sure . .. NO META TAGS KEYWORDS? and if so . . .. what (if anything) should I have in the meta tags?
Technical SEO | | damon1212

0