Indexing non-indexed content and Google crawlers

[Alex-Harford]

On a news website we have a system where articles are given a publish date which is often in the future. The articles were showing up in Google before the publish date despite us not being able to find them linked from anywhere on the website.

I've added a 'noindex' meta tag to articles that shouldn't be live until a future date.

When the date comes for them to appear on the website, the noindex disappears. Is anyone aware of any issues doing this - say Google crawls a page that is noindex, then 2 hours later it finds out it should now be indexed? Should it still appear in Google search, News etc. as normal, as a new page?

Thanks.

[CleverPhD]

Wow!  Nice detective work!   I could see how that one would slip under the radar.

Congrats on finding a needle in a haystack!

You should buy yourself the adult beverage of your choice and have a little toast!

Cheers!

[Alex-Harford]

Good luck and please post back what you find.  This is kind of like one of those "who dun it?" mystery shows! 

Sorted! The link was from a mobile version of the site on an m. subdomain - and only in a facebook share as follows:

[CleverPhD]

I think Screaming Frog has a trial version, I forget if it limits total number of pages etc. as we bought it a while ago.   At least you can try out and see.  May be others who have more tools as well.

[Alex-Harford]

Thanks. I agree I need to get rid of that noindex. The site is new and doesn't have much in the way of tag clouds etc. yet, so it's not like we have a lot of pages to check.

I've used the link: attribute to try and find the offending links each time, but nothing showed up. I use Xenu Link Sleuth rather than Screaming Frog, and I can't find a way to find backlinks with Xenu. Do you know if you can with the free version of Screaming Frog? I've seen the free version described as "almost fully functional" - the number of crawlable links seems to be the main restriction.

[CleverPhD]

I like the automated sitemap answer for the cause (as this has bitten me before), but you mentioned you do not have that.  I would still bet that somewhere on your web site you are linking to the page that you do not want indexed.    It could be a tag cloud page or some other index page.  We had a site that it would accidentally publish out articles on our home page ahead of schedule.  Point here is that when you have a dynamic site with a CMS, you really have to be on your toes with stuff like this as the automation can get you into situations like this.

I would not use the noindex tag and remove it later.  My concern would be that you are sending conflicting signals to Google.  noindex tells good to remove this page from the index.

"When we see the noindex meta tag on a page, Google will completely drop the page from our search results, even if other pages link to it." from GWT

When I read that - it sounds like this is not what you want for this page.

You could also setup your system to show a 404 on the URL until the content is live and then let it 200, but you run into the same issue of Google getting 2 opposite signals on the same page.     Either way, if you first give the signal to Google that you do not want something indexed, you are at the mercy of the next crawl to see if Google looks at it again.

Regardless, you need to get to the crux of the issue, how is Google finding this URL?

I would use a 3rd party spider tool.  We have used Screaming Frog SEO Spider.  There are others out there.  You would be amazed what they find.  The key to this tool is that when it finds something, it also tells you on what page it found it.  We have big sites with thousands of pages and we have used it to find broken links to images and links to pages on our site that now 404.  Really handy to clean things up.  I bet it would find where there is a link on your site that contains the page (or pages) that link to the content.   You can then update that page and not have to worry about using noindex etc.  Also not that the spiders are much better than humans at finding this stuff.  Even if you have looked, the spider looks at things differently.

It also may be as simple as searching for the URL on the web with the link: attribute.  Google may show you where it is finding the link.

Good luck and please post back what you find.  This is kind of like one of those "who dun it?" mystery shows! 

[Alex-Harford]

There is no automated sitemap. We checked every page we could, including feeds.

[Highland]

Do you have an automated sitemap? On at least one occasion, I've found that to be a culprit.

Noindex means it won't be kept in the index. It doesn't mean it won't be crawled. I'm not sure how it would affect crawl timing , tho. I would assume that Google would assume that you would want things not indexed crawled less frequently. Something to potentially try is to use the GWT Fetch as Googlebot tool to force a new crawl of the page and see if that gets it in the index any faster.

http://googlewebmastercentral.blogspot.com/2011/08/submit-urls-to-google-with-fetch-as.html