Why is our noindex tag not working?
-
Hi,
I have the following page where we've implemented a no index tag. But when we run this page in screaming frog or this tool here to verify the noidex is present and functioning, it shows that it's not.
But if you view the source of the page, the code is present in the head tag. And unfortunately we've seen instances where Google is indexing pages we've noindexed. Any thoughts on the example above or why this is happening in Google?
Eddy
-
Hi Eddy,
Edit: this was already answered before I could post my reply. But I've left the example.
The issue with the meta robots tag is that you are using curly quotation marks around robots and noindex:
You have:
“robots**” content=“noindex”/>
Instead of:
name="robots" content="noindex"**/>This will fix your issue.
Cheers,
David
-
That SF response is from the robots.txt block, not a noindex tag though. SF is also ignoring the incorrectly formatted tag (as it should).
Paul
-
The example page does have a noindex tag in place, but it's not formatted correctly, so it's being ignored. Very subtle issue, but your tag is using "smart quotes" around the elements instead of the plain quotation marks that are required for code. If you look very carefully at the page source code, you'll see that they are quotation marks like you'd see in a Word document; the ones at the beginning of robots and noindex curl a different way than the ones at the end.) This usually occurs when the content was written in a word processor instead of a plain-text editor.
Because the tag's not formatted correctly, it's ignored by both the crawling tools and the search engines.
In addition, the site also has all pages blocked from crawling by the sitewide robots.txt file. This and noindex are conflicting instructions to search engines.
If a page is blocked in robots.txt, then the search engine will not crawl the page and so is not able to discover the noindex tag, even if it were formatted correctly. Therefore if the search engine becomes aware of the page in any other way than straight crawling (and there are a number of ways this can happen), then the page will still get indexed.
If it's a dev site, the proper way to keep it from being indexed is to either noindex all pages, or to put the site behind a password so the search engines and public visitors can't access it. If using noindex, the site must not be blocked with a robots.txt directive.
Does that all make sense?
Paul
-
I ran that page thru screaming frog and it came back with a "blocked by robots" status.
The second tool you suggested is not finding the noindex tag and I don't have an explanation for that, nor am I familiar with the tool.
A site command does not return any results.
Are you sure you have a problem? Is there another example you can provide?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Are HTML Sitemaps Still Effective With "Noindex, Follow"?
A site we're working on has hundreds of thousands of inventory pages that are generally "orphaned" pages. To reach them, you need to do a lot of faceting on the search results page. They appear in our XML sitemaps as well, but I'd still consider these orphan pages. To assist with crawling and indexation, we'd like to create HTML sitemaps to link to these pages. Due to the nature (and categorization) of these products, this would mean we'll be creating thousands of individual HTML sitemap pages, which we're hesitant to put into the index. Would the sitemaps still be effective if we add a noindex, follow meta tag? Does this indicate lower quality content in some way, or will it make no difference in how search engines will handle the links therein?
Intermediate & Advanced SEO | | mothner0 -
Can noindexed pages accrue page authority?
My company's site has a large set of pages (tens of thousands) that have very thin or no content. They typically target a single low-competition keyword (and typically rank very well), but the pages have a very high bounce rate and are definitely hurting our domain's overall rankings via Panda (quality ranking). I'm planning on recommending we noindexed these pages temporarily, and reindex each page as resources are able to fill in content. My question is whether an individual page will be able to accrue any page authority for that target term while noindexed. We DO want to rank for all those terms, just not until we have the content to back it up. However, we're in a pretty competitive space up against domains that have been around a lot longer and have higher domain authorities. Like I said, these pages rank well right now, even with thin content. The worry is if we noindex them while we slowly build out content, will our competitors get the edge on those terms (with their subpar but continually available content)? Do you think Google will give us any credit for having had the page all along, just not always indexed?
Intermediate & Advanced SEO | | THandorf0 -
Heading Tags & Content Count
Hi everyone I am looking into this page on our site http://www.key.co.uk/en/key/sack-trucks Just comparing it against competitors in SEMRush, the tool shows a wordcount of this page for over 4089 words, compared with http://www.wickes.co.uk/Wickes-Green-General-Purpose-Sack-Truck-200kg/p/500302 which only has 2658 - it has a lot more written content than our page - where is this word count coming from? Also looking at the same page on our site Woorank suggests we have the word 'sack truck' in the h1 and title too many times - it's only there once, but its this showing because its an exact match keyword? I'm just wondering if there is something wrong with the html or how the page is being crawed?
Intermediate & Advanced SEO | | BeckyKey0 -
Proper Form for Title & Description Tags
Greetings MOZ Community: I operate a real estate web site in New York (www.nyc-officespace-leader.com) that I suspect has been hit by Panda 4.0. I believe a problem is thin content on product pages, which in my case are 350 listing pages. However I am also looking at how title and description tags are formatted for these 350 pages to ensure this is not a factor in the ranking drop. The title descriptions are written like this: <title></span><span class="webkit-html-tag">Flatiron loft for rent | West 21st Street | 1441SF $6604/month</span><span class="webkit-html-tag"></title> Is this sufficiently diverse? Will constantly repeating various street names, square footages and prices work against me? Will Google in a sense consider this thin or repetitive content? It does provide the visitor with key information. The descriptions meta tags are written along these lines: description" content="One of the most desirable full floor sublets in Midtown South. Recent build out, pristine condition, panoramic views, tech chic, spectacular. Top location." /><meta< span=""></meta<> From an SEO perspective are these critical tags written the way they should be? Thanks everyone!! Alan
Intermediate & Advanced SEO | | Kingalan10 -
Canonical tags and product descriptions
I just wanted to check what you guys thought of this strategy for duplicate product descriptions. A sample product is a letter bracelet - a, b, c etc so there are 26 products with identical descriptions. It is going to be extremely difficult to come up with 25 new unique descriptions so with recommendation i'm looking to use the canonical tag. I can't set any to no-index because visitors will look for explicit letters. Because the titles only differ by the letter then a search for either letter bracelet letter a bracelet letter i bracelet will just return results for 'letter bracelet' due to stop words unless the searcher explicitly searches for 'letter "a" bracelet. So I reckon I can make 4 new unique descriptions. I research what are the most popular letters picking 5 from the top (excluding 'a' and 'i'). Equally share the remaining letters between those 5 and with each group set a canonical tag pointing to the primary letter of that group. Does this seem a sensible thing to do?
Intermediate & Advanced SEO | | MickEdwards0 -
Should we use the rel-canonical tag?
We have a secure version of our site, as we often gather sensitive business information from our clients. Our https pages have been indexed as well as our http version. Could it still be a problem to have an http and an https version of our site indexed by Google? Is this seen as being a duplicate site? If so can this be resolved with a rel=canonical tag pointing to the http version? Thanks
Intermediate & Advanced SEO | | annieplaskett1 -
Which Authorship Strategy Works?
We want to claim our articles and get our picture next to our articles in the search engines. I was offered this article http://www.virante.com/blog/2012/01/08/how-to-show-your-author-photo-in-google-search-results/ but Google has this article: http://support.google.com/webmasters/bin/answer.py?hl=en&answer=1408986 Google's way is simpler, Is that all I need to do? This is for a Joomla site Thanks!
Intermediate & Advanced SEO | | BobGW0 -
Original Source and Canonical tags
We've been using canonical links to protect site SEO for contributor content and requiring canonical of our partners (as well as tagging internal duplicate content with canonical). Most other media sites have been doing the same but this is a moving target. I'm now hearing that the original source tag is now a better option. Special focus for us is placement on google news. Any guidance?
Intermediate & Advanced SEO | | jbertfield0