Why is our noindex tag not working?
-
Hi,
I have the following page where we've implemented a no index tag. But when we run this page in screaming frog or this tool here to verify the noidex is present and functioning, it shows that it's not.
But if you view the source of the page, the code is present in the head tag. And unfortunately we've seen instances where Google is indexing pages we've noindexed. Any thoughts on the example above or why this is happening in Google?
Eddy
-
Hi Eddy,
Edit: this was already answered before I could post my reply. But I've left the example.
The issue with the meta robots tag is that you are using curly quotation marks around robots and noindex:
You have:
“robots**” content=“noindex”/>
Instead of:
name="robots" content="noindex"**/>This will fix your issue.
Cheers,
David
-
That SF response is from the robots.txt block, not a noindex tag though. SF is also ignoring the incorrectly formatted tag (as it should).
Paul
-
The example page does have a noindex tag in place, but it's not formatted correctly, so it's being ignored. Very subtle issue, but your tag is using "smart quotes" around the elements instead of the plain quotation marks that are required for code. If you look very carefully at the page source code, you'll see that they are quotation marks like you'd see in a Word document; the ones at the beginning of robots and noindex curl a different way than the ones at the end.) This usually occurs when the content was written in a word processor instead of a plain-text editor.
Because the tag's not formatted correctly, it's ignored by both the crawling tools and the search engines.
In addition, the site also has all pages blocked from crawling by the sitewide robots.txt file. This and noindex are conflicting instructions to search engines.
If a page is blocked in robots.txt, then the search engine will not crawl the page and so is not able to discover the noindex tag, even if it were formatted correctly. Therefore if the search engine becomes aware of the page in any other way than straight crawling (and there are a number of ways this can happen), then the page will still get indexed.
If it's a dev site, the proper way to keep it from being indexed is to either noindex all pages, or to put the site behind a password so the search engines and public visitors can't access it. If using noindex, the site must not be blocked with a robots.txt directive.
Does that all make sense?
Paul
-
I ran that page thru screaming frog and it came back with a "blocked by robots" status.
The second tool you suggested is not finding the noindex tag and I don't have an explanation for that, nor am I familiar with the tool.
A site command does not return any results.
Are you sure you have a problem? Is there another example you can provide?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
International Blog Structure & Hreflang Tags
Hi all, I'm running an international website across 5 regions using a correct hreflang setup. A problem I think I have is that my blog structure is not standardized and also uses hreflang tags for each blog article. This has naturally caused Google to index each of the pages across each region, meaning a massive amount of pages are being crawled. I know hreflang solves and issues with duplication penalties, but I have another question. If I have legacy blog articles that are considered low quality by Google, is that counting against my site once or multiple times for each time the blog is replicated across each region? I'm not sure if hreflang is something that would tell Google this. For example, if I have low quality blog posts: blog/en-us/low-quality-article-1
Intermediate & Advanced SEO | | MattBassos
blog/en-gb/low-quality-article-1
blog/en-ca/low-quality-article-1 Do you think Google is counting this as 3 low quality articles or just 1 if hreflang is correctly implemented? Any insights would be great because I'm considering to cull the international setup of the blog articles and use just /blog across each region.0 -
Alt tags seo boost
Do alt tags really give you a boost in terms of ranking or is it more for ranking in google images ? Thank you,
Intermediate & Advanced SEO | | seoanalytics0 -
SEO Title Versus Meta Description Tag
From an SEO perspective, is the title tag more important than the description tag? We use a set format for these tags on our real estate web site. The site contains 300 listings. Sample Title Tag:
Intermediate & Advanced SEO | | Kingalan1
Greenwich Village | Office Space Rental| 2300SF $9583/month Sample Description Tag:
Classic Greenwich Village office rental. Hardwood floors, 11' ceiling. 5 oversized windows. 24/7 attended lobby. Renovated common areas. Below market rent. Are we shooting ourselves in the foot by repeating the Square Footage and monthly rent amounts in the title tag? Should this tag be used for a short more descriptive terms so as to maximize the SEO benefit? Should these numbers be listed in the description tag? The listings are not heavily SEO optimized so I don't know whether this is really a non-issue.0 -
Two pages on same domain - Is this a proper use of the canonical tag?
I have a domain with two pages in question--one is an article with 2,000 words and the other is a FAQ with 300 words. The 300 word FAQ is copied, word-for-word and pasted inside of the 2,000 word article. Would it be a proper use of the canonical tag to point the smaller, 300 word FAQ at the 2,000 word article? Since the 300 word article is identical to a portion of the 2,000 word article, will Google see this as duplicate content? Thanks in advance for any helpful insight.
Intermediate & Advanced SEO | | andrewv0 -
Canonical Tag for Pages with Less Content
I am considering using a cross-domain canonical tag for pages that are very similar but one has less content than the other. The domains are geo specific, so for example. www.page.com - with content xxx, yyy, zzz, and www.page.fr with content xxx is this a problem because while there is clearly duplicate content here the pages are not actually significantly similar since there is so much less content on one page than the other?
Intermediate & Advanced SEO | | theLotter0 -
Canonical Tags?
I read that Google will "honor" these tags if your website has two url's with duplicate content. The duplicate content does not show up in my SEOmoz crawls report but they do in the search engines and many of "non authoritative links" that are generated from my search feature j(ugly url's with % ...not real user friendly) are ranking higher than the "good URL" links. So if I do the canonical tags I guess my higher ranking bad urls will drop. I even read that google might even completely overlook the links. I read somewhere that the best way to do this is with a 301 redirect...is that correct? I m ranking pretty good with my main keyword terms so I am afraid to make changes not knowing the effect. Any suggestions? Thanks, Boo
Intermediate & Advanced SEO | | Boodreaux0 -
Why Put an H1 Tag On A Product?
Why would you put an H1 tag on a product name? I came across this in another forum and thought I'd float it here.
Intermediate & Advanced SEO | | AWCthreads0 -
Why should your title and H1 tag be different?
Is it dangerous to have your H1 tag and your title the exact same thing? My thought was that it's not be the best use of space, but that it couldn't cause harm. What do you think?
Intermediate & Advanced SEO | | MarieHaynes7