Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

Meta Robots Noindex and Robots.txt File

Technical SEO

473

iskq last edited by

Can we use meta robots noindex and robots.txt together?
1 Reply Last reply
Reply Quote 0
Devanur-Rafi last edited by

Hi, let us take a scenario. test.html has been blocked using robots.txt file. In this scenario, the bots that respect the robots exclusion protocol will not crawl the page to encounter the page level robots noindex attribute. So does not make any sense to use both the page level robots exclusion method and server side robots.txt file.

Here you go for more:

http://support.google.com/webmasters/bin/answer.py?hl=en&answer=93708

And here:

http://support.google.com/webmasters/bin/answer.py?hl=en&answer=93710

Best regards,

Devanur Rafi.
1 Reply Last reply
Reply Quote 2

Got a burning SEO question?

Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.

Start my free trial

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

Meta Description for ALL my Posts is Being Overwritten!

Every single one of my posts meta descriptions are being overritten with the same meta description! I use Yoast and have proper meta descriptions in my posts, but something seems to be overriding them. https://ibb.co/rcgWNXb Any idea what could be causing this? Thank you!!! Mike
Technical SEO | | naturalsociety

1
Site Hack In Meta Description

Hey MOZ Community, I am looking for some help in identifying where the following meta description is coming from on this home page - https://www.apins.com. I have scrubbed through the page source without being able to locate where the content is being pulled from. The website is built on WordPress and metas were updated using Yoast, but I am wondering if an installed plugin could be the culprit. On top of this, I have had a developer take a look for the "hack" and they have assured that the issue has been removed. I have submitted the URL in GSC a couple of times to be re-indexed but have not had much luck. Any thoughts would be much appreciated, the displayed description is below. The health screening plays http://buyviagraonlineccm.com/ a significant and key role in detecting potentially life-threatening illnesses such as cancer, heart ...
Technical SEO | | jordankremer

0
Do I need to block my cart page in robots.txt?

I have a site with woocommerce. Do I need to block the cart page?
Technical SEO | | EcommerceSite

0
The use of robots.txt

Could someone please confirm that if I do not want to block any pages from my URL, then I do not need a robots.txt file on my site? Thanks
Technical SEO | | ICON_Malta

0
How many times robots.txt gets visited by crawlers, especially Google?

Hi, Do you know if there's any way to track how often robots.txt file has been crawled? I know we can check when is the latest downloaded from webmaster tool, but I actually want to know if they download every time crawlers visit any page on the site (e.g. hundreds of thousands of times every day), or less. thanks...
Technical SEO | | linklater

0
Internal search : rel=canonical vs noindex vs robots.txt

Hi everyone, I have a website with a lot of internal search results pages indexed. I'm not asking if they should be indexed or not, I know they should not according to Google's guidelines. And they make a bunch of duplicated pages so I want to solve this problem. The thing is, if I noindex them, the site is gonna lose a non-negligible chunk of traffic : nearly 13% according to google analytics !!! I thought of blocking them in robots.txt. This solution would not keep them out of the index. But the pages appearing in GG SERPS would then look empty (no title, no description), thus their CTR would plummet and I would lose a bit of traffic too... The last idea I had was to use a rel=canonical tag pointing to the original search page (that is empty, without results), but it would probably have the same effect as noindexing them, wouldn't it ? (never tried so I'm not sure of this) Of course I did some research on the subject, but each of my finding recommanded one of the 3 methods only ! One even recommanded noindex+robots.txt block which is stupid because the noindex would then be useless... Is there somebody who can tell me which option is the best to keep this traffic ? Thanks a million
Technical SEO | | JohannCR

0
Long Meta Descriptions

I want to create a template for Meta titles, descriptions and keywords on my website for old news and minor pages in order to get some long tail traffic from them. The only template I can think to use for the descriptions takes the first sentence of the news article (which often if above 160 characters). Since these are minor pages, how big of a problem is that? Thanks!
Technical SEO | | theLotter

0
Robots.txt not working?

Hello This is my robots.txt file http://www.theprinterdepo.com/Robots.txt However I have 8000 warnings on my dashboard like this:4 What am I missing on the file¿ Crawl Diagnostics Report On-Page Properties <dl> <dt>Title</dt> <dd>Not present/empty</dd> <dt>Meta Description</dt> <dd>Not present/empty</dd> <dt>Meta Robots</dt> <dd>Not present/empty</dd> <dt>Meta Refresh</dt> <dd>Not present/empty</dd> </dl> URL: http://www.theprinterdepo.com/catalog/product_compare/add/product/100/uenc/aHR0cDovL3d3dy50aGVwcmludGVyZGVwby5jb20vaHAtbWFpbnRlbmFjZS1raXQtZm9yLTQtbGo0LWxqNS1mb3ItZXhjaGFuZ2UtcmVmdWJpc2hlZA,,/ 0 Errors No errors found! 1 Warning 302 (Temporary Redirect) Found about 5 hours ago <a class="more">Read More</a>
Technical SEO | | levalencia1

0