Noindex,follow - linked pages not showing

[Nobody1556905035114]

We have a blog on our site where the homepage and category pages have "noindex,follow" but the articles have "index,follow".

Recently we have noticed that the article pages are no longer showing in the Google SERPs (but they are in Bing!) - this was done by using the "site:" search operator.

Have double-checked our robots.txt file too just in case something silly had slipped in, but that's as it should be...

Has anyone else noticed similar behaviour or could suggest things I could check?

Thanks!

[AllenSL]

Well you're on Wordpress and are using YoastSEO. When a Wordpress category is created, a URL is generated for that category.

Your sitemap was created with Yoast:

Sitemap	Last Modified

|

• https://tipbox.abcam.com/post-sitemap.xml

|

• 2018-08-23 08:10 +01:00

• https://tipbox.abcam.com/page-sitemap.xml

|

• 2018-08-23 08:21 +01:00

• https://tipbox.abcam.com/attachment-sitemap.xml

|

• 2018-08-23 08:08 +01:00

• https://tipbox.abcam.com/life_work-sitemap.xml

|

• 2018-08-07 10:40 +01:00

• https://tipbox.abcam.com/category-sitemap.xml

|

• 2018-08-23 08:10 +01:00

• https://tipbox.abcam.com/author-sitemap.xml

|

• 2018-08-23 08:13 +01:00

|

I can see your articles are indexed now, but I would still recommend removing the Wordpress category URL's from your sitemap. Since the sitemap is commonly used for the things you want Google to crawl and index, I would add the article urls and content with "index,follow" webpages directly to your xml sitemap instead of linking the category pages you don't want indexed.

(IE: ie: http://www.genetex.com/sitemap.xml)

Yoast should give you this option in the settings for xml sitemap generation. If not, I would recommend using Screaming Frog to generate the sitemap.

[Nobody1556905035114]

Just as an update to this question, I submitted XML sitemaps directly to the blog articles and those pages are still not showing in the Google SERPs. It seems that new pages are discovered quite quickly (as per Google Alerts) but are then dropped from the index within a day or so.

The only pages which are returned consistently are links to the page which allows comments to be added.

The links which were initially identified as broken, were not actually broken so there was nothing to fix there.

Next step I can think of is to attempt some page sculpting by setting a noindex on the comments pages...

If anyone has any more thoughts or ideas, I'd appreciate your input

[Nobody1556905035114]

Great - thanks for your help

[Nobody1556905035114]

I resubmitted the sitemap for the blog in GWT and no errors were found...

I have to say I am v surprised at the number of dead links - we don't have that many blog posts so unless this is indicating content on our main site (where the pages are still .  Even then, as I mentioned to Alan, the only missing content Google Webmaster Tools picks up on is where event tracking is used and it thinks the label is a link.... I did ask Google about these erroneous missing page and they said there was nothing that can be done to indicate they're not meant to be pages and that it would not affect the site's quality.

BTW, An article we published a few hours ago is now showing up in the Google results so it does seem like the rest of the pages have been penalised

Time to figure out what's going on with the missing pages...

Thanks, Irving

[AlanMosley]

i sent the list, i had a bit of a look and it may be that they were timing out

[Nobody1556905035114]

Thanks Alan, have DM'ed you.

[irvingw]

Submit a sitemap.xml file for these pages you want indexed, If they are linked to on the site and not blocked in robots.txt they will get indexed again. Definitely fix that sick amount of broken links, Google could be determining that these pages are not worth anything because the links on them are all dead ends.

[AlanMosley]

The broken links were found using the Bing api. so bing will see them as such,

If yougive me a email, i willl send you the list

[Nobody1556905035114]

39 no-index pages on the blog could be correct with the category pages.

I'm quite surprised at the number of broken links - is this specific to /blog and are they actual links? GWT usually picks up event tracking as broken links...

Good point about the homepage - I should get a canonical tag on that...

Thanks!

[AlanMosley]

I found 39 pages that have been no-index, does that add up?

I also found 33,000 broken links.

anouther problem you have is that both http://www.abcam.com/blog/ and http://www.abcam.com/blog/index.cfm are linked to in your site, this means that the pagerank is split. you should link to only http://www.abcam.com/blog/

[Nobody1556905035114]

The blog homepage is http://www.abcam.com/blog

@Alan: The rest of the site is indexable, just the the blog area where noindex has been used (the blog homepage and category pages are auto-generated and repeat a lot of the content in the articles)

@Shailendra: Yes, they were indexed - the last Google Alert which specifically highlights content from the blog is mid-June.

[IM_Learner]

Firstly, you don't need to write index,follow on normal pages. Secondly, as you say, "no longer showing in Google SERPs", this means that it was earlier indexed, right? Now if it is no longer in Google's index, it means penalization. Please give the url of your website.

[AlanMosley]

It may have something to do with the homepage being noindex, as that is unusual.

Can we get a url, I may find what you missed?

[SanketPatel]

Hi,

Can you please share URL ?