Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
Why is our noindex tag not working?
-
Hi,
I have the following page where we've implemented a no index tag. But when we run this page in screaming frog or this tool here to verify the noidex is present and functioning, it shows that it's not.
But if you view the source of the page, the code is present in the head tag. And unfortunately we've seen instances where Google is indexing pages we've noindexed. Any thoughts on the example above or why this is happening in Google?
Eddy
-
Hi Eddy,
Edit: this was already answered before I could post my reply. But I've left the example.
The issue with the meta robots tag is that you are using curly quotation marks around robots and noindex:
You have:
“robots**” content=“noindex”/>
Instead of:
name="robots" content="noindex"**/>This will fix your issue.
Cheers,
David
-
That SF response is from the robots.txt block, not a noindex tag though. SF is also ignoring the incorrectly formatted tag (as it should).
Paul
-
The example page does have a noindex tag in place, but it's not formatted correctly, so it's being ignored. Very subtle issue, but your tag is using "smart quotes" around the elements instead of the plain quotation marks that are required for code. If you look very carefully at the page source code, you'll see that they are quotation marks like you'd see in a Word document; the ones at the beginning of robots and noindex curl a different way than the ones at the end.) This usually occurs when the content was written in a word processor instead of a plain-text editor.
Because the tag's not formatted correctly, it's ignored by both the crawling tools and the search engines.
In addition, the site also has all pages blocked from crawling by the sitewide robots.txt file. This and noindex are conflicting instructions to search engines.
If a page is blocked in robots.txt, then the search engine will not crawl the page and so is not able to discover the noindex tag, even if it were formatted correctly. Therefore if the search engine becomes aware of the page in any other way than straight crawling (and there are a number of ways this can happen), then the page will still get indexed.
If it's a dev site, the proper way to keep it from being indexed is to either noindex all pages, or to put the site behind a password so the search engines and public visitors can't access it. If using noindex, the site must not be blocked with a robots.txt directive.
Does that all make sense?
Paul
-
I ran that page thru screaming frog and it came back with a "blocked by robots" status.
The second tool you suggested is not finding the noindex tag and I don't have an explanation for that, nor am I familiar with the tool.
A site command does not return any results.
Are you sure you have a problem? Is there another example you can provide?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
No index detected in robots meta tag GSC issue_Help Please
Hi Everyone, We just did a site migration ( URL structure change, site redesign, CMS change). During migration, dev team messed up badly on a few things including SEO. The old site had pages canonicalized and self canonicalized <> New site doesn't have anything (CMS dev error) so we are working retroactively to add canonicalization mechanism The legacy site had URL’s ending with a trailing slash “/” <> new site got redirected to Set of url’s without “/” New site action : All robots are allowed: A new sitemap is submitted to google search console So here is my problem (it been a long 24hr night for me 🙂 ) 1. Now when I look at GSC homepage URL it says that old page is self canonicalized and currently in index (old page with a trailing slash at the end of URL). 2. When I try to perform a live URL test, I get the message "No: 'noindex' detected in 'robots' meta tag" , so indexation cant be done. I have no idea where noindex is coming from. 3. Robots.txt in search console still showing old file ( no noindex there ) I tried to submit new file but old one still coming up. When I click on "See live robots.txt" I get current robots. 4. I see that old page is still canonicalized and attempting to index redirected old page might be confusing google Hope someone can help to get the new page indexed! I really need it 🙂 Please ping me if you need more clarification. Thank you ! Thank you
Intermediate & Advanced SEO | | bgvsiteadmin1 -
Title tag: Long tail words or keyword dilution?
Hi all, I am a newbie to SEO. Lately, I have been struggling to optimize my title tag. Ones say that we should have long tail words in title tags because long tail words improve click through rate and generate quality leads. On the other hand, ones say that putting other words in the title tag will dilute the main keyword that my page ranks for. Do keywords really dilute each other in the title tags? I am really confused. Let me give this an example: Web Design Services | Company Name Web Design Services with Conversion Focused | Company Name Which one would you prefer and why? Thank you. 😄 Best, Raymond
Intermediate & Advanced SEO | | Raymondlee0 -
Lowercase VS. Uppercase Canonical tags?
Hi MOZ, I was hoping that someone could help shed some light on an issue I'm having with URL structure and the canonical tag. The company I work for is a distributor of electrical products and our E-commerce site is structured so that our URL's (specifically, our product detail page URL's) include a portion (the part #) that is all uppercase (e.g: buy/OEL-Worldwide-Industries/AFW-PG-10-10). The issue is that we have just recently included a canonical tag in all of our product detail pages and the programmer that worked on this project has every canonical tag in lowercase instead of uppercase. Now, in GWT, I'm seeing over 20,000-25,000 "duplicate title tags" or "duplicate descriptions". Is this an issue? Could this issue be resolved by simply changing the canonical tag to reflect the uppercase URL's? I'm not too well versed in canonical tags and would love a little insight. Thanks!
Intermediate & Advanced SEO | | GalcoIndustrial0 -
Should I noindex the site search page? It is generating 4% of my organic traffic.
I read about some recommendations to noindex the URL of the site search.
Intermediate & Advanced SEO | | lcourse
Checked in analytics that site search URL generated about 4% of my total organic search traffic (<2% of sales). My reasoning is that site search may generate duplicated content issues and may prevent the more relevant product or category pages from showing up instead. Would you noindex this page or not? Any thoughts?0 -
Noindex a meta refresh site
I have a client's site that is a vanity URL, i.e. www.example.com, that is setup as a meta refresh to the client's flagship site: www22.example.com, however we have been seeing Google include the Vanity URL in the index, in some cases ahead of the flagship site. What we'd like to do is to de-index that vanity URL. We have included a no-index meta tag to the vanity URL, however we noticed within 24 hours, actually less, the flagship site also went away as well. When we removed the noindex, both vanity and flagship sites came back. We noticed in Google Webmaster that the flagship site's robots.txt file was corrupt and was also in need of fixing, and we are in process of fixing that - Question: Is there a way to noindex vanity URL and NOT flagship site? Was it due to meta refresh redirect that the noindex moved out the flagship as well? Was it maybe due to my conducting a google fetch and then submitting the flagship home page that the site reappeared? The robots.txt is still not corrected, so we don't believe that's tied in here. To add to the additional complexity, the client is UNABLE to employ a 301 redirect, which was what I recommended initially. Anyone have any thoughts at all, MUCH appreciated!
Intermediate & Advanced SEO | | ACNINTERACTIVE0 -
Should I remove Meta Keywords tags?
Hi, Do you recommend removing Meta Keywords or is there "nothing to lose" with having them? Thanks
Intermediate & Advanced SEO | | BeytzNet0 -
Why Put an H1 Tag On A Product?
Why would you put an H1 tag on a product name? I came across this in another forum and thought I'd float it here.
Intermediate & Advanced SEO | | AWCthreads0 -
How Rel=Prev & Rel=Next work for me?
I have implemented Rel=Prev & Rel=Next tag on my website. I would like to give example URL to know more about it. http://www.vistapatioumbrellas.com/market-umbrellas?limit=40&p=3 http://www.vistapatioumbrellas.com/market-umbrellas?limit=40&p=4 http://www.vistapatioumbrellas.com/market-umbrellas?limit=40&p=5 Right now, I have blocked paginated pages by Robots.txt by following query. Disallow: /*?p= I have removed disallow syntax from Robots.txt for paginated pages. But, I have confusion with duplicate page title. If you will check all 3 pages so you will find out duplicate page title across all pages. I know that, duplicate page title is harmful for SEO. Will Google crawl + index all paginated pages? If yes so which page will get maximum benefits in organic ranking? Is there any specific way which may help me to solve this issue?
Intermediate & Advanced SEO | | CommercePundit0