NoIndex/NoFollow pages showing up when doing a Google search using "Site:" parameter
-
We recently launched a beta version of our new website in a subdomain of our existing site. The existing site is www.fonts.com with the beta living at new.fonts.com. We do not want Google to crawl the new site until it's out of beta so we have added the following on all pages:
However, one of our team members noticed that google is displaying results from new.fonts.com when doing an "site:new.fonts.com" search (see attached screenshot). Is it possible that Google is indexing the content despite the noindex, nofollow tags? We have double checked the syntax and it seems correct except the trailing "/". I know Google still crawls noindexed pages, however, the fact that they're showing up in search results using the site search syntax is unsettling.
Any thoughts would be appreciated!
-
Thanks, appreciate you taking the time to write out a response!
-
Thank you for your reply. I will get this information over to the dev team!
-
Hi Chris
If Google sees a link to the page it may still list it in its index even though when they got there they saw the noindex tag so they didn't crawl it.
The rational is they see a link from your main site with some anchor text and index the link based on the anchor text they can't crawl it because you say not to, but they still have some information about the page from your anchor text.
Here is a direct Matt Cutts Quote:
"Our highest duty has to be to our users, not to an individual webmaster. When a user does a navigational query and we don’t return the right link because of a NOINDEX tag, it hurts the user experience (plus it looks like a Google issue). If a webmaster really wants to be out of Google without even a single trace, they can use Google’s url removal tool."
REF: http://www.mattcutts.com/blog/google-noindex-behavior/
You can block access to the test site (which is what we do) via htacess (if you're on a Linux Server) and use the Google Index Removal Tool to strip out the currently indexed pages.
I hope that helps.
-
If you have nofollow on all the pages, there is a chance it is being caused because google can't follow any links to your pages tho crawl and update them with the no-index tag.
Try changing your links to noindex, follow.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Spammy structured data for http://www.heritageprinting.com/ might be dropped from search results
We received the above message, which I'm see may also have. Before I go making hours of edits can someone give me an opinion on what may need fixed? Here's a link to one of our products: http://heritageprinting.com/products/step-and-repeat.phpAll products are uniquely marked upIt may be the $ dollar sign, but I'm not certain.Looking at WMT > Search Appearance > Structured Data, I see no errors for Schema Markup. TY in advance :)KJr
Technical SEO | | KevnJr0 -
From: http://www. to https://
Hi all, I am changing my hosting for legal and SEO reasons from http://www to https:// . Now I hear different stories on the redirects: 1: should i try and change my backlinks? 2: internally all links will be 301 redirected at first. Than I want to (manually) change them. It;s within Wordpress so there should be a plugin for this. Tips? 3: Will it affect my rankings and for what period? What I now know that at first it will drop little but eventually you will rank higher than before. Thanks so much in advance! Tymen
Technical SEO | | Tymen1 -
How do I deal with /mobile/ page after responsive re-design?
Hi guys, One of our clients used to have a website that would redirect mobile traffic to a /mobile/ page. Thankfully we've finally gone fully responsive and there is no need for this /mobile/ page. Trouble is, www.clientsite.com.au**/mobile/** is still in the Google index and going to a 404 right now. What is the best way to deal with it? Should we be 301 redirecting /mobile/ to / (the home page)? Would be most grateful for any ideas. Thanks!
Technical SEO | | WCR0 -
How could you make a URL/Breadcrumb structure appear different in Google than when you click into site?
I'm seeing a competitor be able to make their URL/Breadcrumb stucture appear different in Google than on the site. Google shows a 3-4 category silo for the page but once clicked the page is off root. How could you do this?
Technical SEO | | TicketCity0 -
New Page Showing Up On My Reports w/o Page Title, Words, etc - However, I didn't create it
I have a WordPress site and I was doing a crawl for errors and it is now showing up as of today that this page : https://thinkbiglearnsmart.com/event-registration/?event_id=551&name_of_event=HTML5 CSS3 is new and has no page title, words, etc. I am not even sure where this page or URL came from. I was messing with the robots.txt file to allow some /category/ posts that were being hidden, but I didn't re-allow anything with the above appendages. I just want to make sure that I didn't screw something up that is now going to impact my rankings - this was just a really odd message to come up as I didn't create this page recently - and that shouldnt even be a page accessible to the public. When I edit the page - it is using an Event Espresso (WordPress plugin) shortcode - and I don't want to noindex this page as it is all of my events. Sorry this post is confusing, any help or insight would be appreciated! I am also interested in hiring someone for some hourly consulting work on SEO type issues if anyone has any references. Thank you!
Technical SEO | | webbmason0 -
How Does Google's "index" find the location of pages in the "page directory" to return?
This is my understanding of how Google's search works, and I am unsure about one thing in specific: Google continuously crawls websites and stores each page it finds (let's call it "page directory") Google's "page directory" is a cache so it isn't the "live" version of the page Google has separate storage called "the index" which contains all the keywords searched. These keywords in "the index" point to the pages in the "page directory" that contain the same keywords. When someone searches a keyword, that keyword is accessed in the "index" and returns all relevant pages in the "page directory" These returned pages are given ranks based on the algorithm The one part I'm unsure of is how Google's "index" knows the location of relevant pages in the "page directory". The keyword entries in the "index" point to the "page directory" somehow. I'm thinking each page has a url in the "page directory", and the entries in the "index" contain these urls. Since Google's "page directory" is a cache, would the urls be the same as the live website (and would the keywords in the "index" point to these urls)? For example if webpage is found at wwww.website.com/page1, would the "page directory" store this page under that url in Google's cache? The reason I want to discuss this is to know the effects of changing a pages url by understanding how the search process works better.
Technical SEO | | reidsteven750 -
Choosing the right page for rel="canonical"
I am wondering how you would choose which page to use as a canonical ? All our articles sit in an article section and they are called in the url when linked from a particular category. Since some articles are in many categories, we may have several links for the same page. My first idea was to put the one in the article category as the canonical, but I wonder if Google will lose the context of the page for it's ranking because it will not be in the proper category. For exemple, this page in the article section : http://www.bdc.ca/en/advice_centre/articles/Pages/exporting_entering.aspx Same page in the Expand Your Sales > Going Global section : http://www.bdc.ca/EN/advice_centre/expand_your_sales/going_global_or_international_markets/Pages/RelatedArticles.aspx?PATH=/EN/advice_centre/articles/Pages/exporting_entering.aspx The second one has much more context related to it, like the breadcrumb is showing the path and the left menu is open at the right place. For this example, I would choose te second one, but some articles may be found in 2 or 3 categories. If you could share your lights on this it would be very appreciated ! Thanks
Technical SEO | | jfmonfette0 -
Google Search Parameters
Couple quick questions. Is using the parameter pws=0 still useful for turning off personalization? Is there a way to set my location as a URL parameter as well? For instance, I want to set my location to United States, can this be done with a URL param the same way as pws=0?
Technical SEO | | nbyloff0
