Why is our noindex tag not working?
-
Hi,
I have the following page where we've implemented a no index tag. But when we run this page in screaming frog or this tool here to verify the noidex is present and functioning, it shows that it's not.
But if you view the source of the page, the code is present in the head tag. And unfortunately we've seen instances where Google is indexing pages we've noindexed. Any thoughts on the example above or why this is happening in Google?
Eddy
-
Hi Eddy,
Edit: this was already answered before I could post my reply. But I've left the example.
The issue with the meta robots tag is that you are using curly quotation marks around robots and noindex:
You have:
“robots**” content=“noindex”/>
Instead of:
name="robots" content="noindex"**/>This will fix your issue.
Cheers,
David
-
That SF response is from the robots.txt block, not a noindex tag though. SF is also ignoring the incorrectly formatted tag (as it should).
Paul
-
The example page does have a noindex tag in place, but it's not formatted correctly, so it's being ignored. Very subtle issue, but your tag is using "smart quotes" around the elements instead of the plain quotation marks that are required for code. If you look very carefully at the page source code, you'll see that they are quotation marks like you'd see in a Word document; the ones at the beginning of robots and noindex curl a different way than the ones at the end.) This usually occurs when the content was written in a word processor instead of a plain-text editor.
Because the tag's not formatted correctly, it's ignored by both the crawling tools and the search engines.
In addition, the site also has all pages blocked from crawling by the sitewide robots.txt file. This and noindex are conflicting instructions to search engines.
If a page is blocked in robots.txt, then the search engine will not crawl the page and so is not able to discover the noindex tag, even if it were formatted correctly. Therefore if the search engine becomes aware of the page in any other way than straight crawling (and there are a number of ways this can happen), then the page will still get indexed.
If it's a dev site, the proper way to keep it from being indexed is to either noindex all pages, or to put the site behind a password so the search engines and public visitors can't access it. If using noindex, the site must not be blocked with a robots.txt directive.
Does that all make sense?
Paul
-
I ran that page thru screaming frog and it came back with a "blocked by robots" status.
The second tool you suggested is not finding the noindex tag and I don't have an explanation for that, nor am I familiar with the tool.
A site command does not return any results.
Are you sure you have a problem? Is there another example you can provide?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Have issues about expired domain? How to work on it? How to redirect?
Using expired domains to redirect on your site? For this you need to first Active your expired domainsand put two to three articles After google will got it that its now active and so google start crawling after that you need to put same articles that you created on expired site on your new site with same urls Now make a wild card redirect to exact urls or posts you created on your site
Intermediate & Advanced SEO | | LyleTLai0 -
Working on new link structure
Hello Mozzzzzzzzzzz I'm currently working on the new link structure for our website. We currently organize our content in sub folder =Main category
Intermediate & Advanced SEO | | floaumet
= = Sub category
= = =Specialty
= = = Product main name
= = = ==Product specific name
= = = == =Manufacturer Each of them has some potential strong KW and I will be happy to use it on the URL. URL are more than 50 kw long when I use all This are very niche item for which people may refer to them through different names (potential folders) My current concerns will be Should we make long url respecting the structure (Main category)/(Sub category)/(Specialty)/(Product main name)/(Product specific name)/(maufacturer) Should I combine some like (Main category)/(Sub category)/(Specialty)/(Product main name)/(Product specific name)-(maufacturer) Should I keep them simple /(product_main name) Should I keep the main folders just to display the articles belonging to this category (Main category)/(Sub category)/(Specialty)/(Product main name) and then keep the product under a sub folder only? Any other idea?0 -
H2 Tags- Can you have more than 1 H2 tag
Hi All, Screaming frog has identified that we have a few H2 tags on our pages , although we only have 1 H1 tag. We have numerous H3,H4's etc. I am wondering, is it good SEO to have only 1 H2 tag like with H1 tag or can you have more ? thanks Peter
Intermediate & Advanced SEO | | PeteC120 -
Will Google recognize a canonical to a re-directed URL works?
A third party canonicalizes to our content, and we've recently needed to re-direct that content to a new URL. The third party is going to take some time updating their canonicals, and I am wondering if search engines will still recognize the canonical even though there is a re-direct in place?
Intermediate & Advanced SEO | | nicole.healthline0 -
Cross Domain Rel Canonical tags vs. Rel Canonical Tags for internal webpages
Today I noticed that one of my colleagues was pointing rel canonical tags to a third party domain on a few specific pages on a client's website. This was a standard rel canonical tag that was written Up to this point I haven't seen too many webmasters point a rel canonical to a third party domain. However after doing some reading in the Google Webmaster Tools blog I realized that cross domain rel canonicals are indeed a viable strategy to avoid duplicate content. My question is this; should rel canonical tags be written the same way when dealing with internal duplicate content vs. external duplicate content? Would a rel=author tag be more appropriate when addressing 3rd party website duplicate content issues? Any feedback would be appreciated.
Intermediate & Advanced SEO | | VanguardCommunications0 -
Company name in title tags for lesser known brands - yes or no?
Hi Mozzers I read an interesting post over on Authority Labs this morning about title tag length and how Google changes the way they are displayed. The author Brian advises that, "if you want your title tag to remain unchanged, it's worth making sure that you're staying within the 50-59 character window and that your titles fit with the content of the page". This got me thinking... Given the limited amount of title tag characters that are now shown in the SERPs, I find it difficult to include a primary keyword, secondary keyword and the company name. So, if you're a lesser known brand is it worth sacrificing your company name in the title tags of deeper pages for a secondary keyword to help with rankings, or even a special offer to grab a users eye in the SERPs? What are people's views on this? Thanks Anthony
Intermediate & Advanced SEO | | Tone_Agency0 -
Anchor Tag around Table / Block
Our homepage (here) has four large promotional sections taking up most of the real estate. Each promo section has an image and styled text. We want each promo section to link to the appropriate page, so we created the promo sections as and wrapped each in an anchor. That works fine for users but I tried viewing our site in a text-only browser (Lynx) and couldn't follow those links! My fear is that GoogleBot can't follow them either and doesn't know what anchor text to pull. So, my question: What's the best way to make this entire block clickable, but still have it crawlable by robots? Or is our current implementation ok? For reference, here's a simplified version of the relevant code block: |
Intermediate & Advanced SEO | | Richline_Digital
|
All Diamonds
Extra 20% Off
|
[| 
|
Jessica Simspon
Extra 20% Off
|](http://jessicasimpson.jewelry.com/shop/)
0 -
H1 Tags and HTML5
I have read that now you can have multiple h1 tags on a page without it negatively impacting SEO. Previously it was advised to only have 1 h1 tag on a page. Example: with the new semantic mark up you could have separate h1 tags for the header, article, aside and footer. Is this really the case?
Intermediate & Advanced SEO | | bronxpad0