Noindex,follow - linked pages not showing
-
We have a blog on our site where the homepage and category pages have "noindex,follow" but the articles have "index,follow".
Recently we have noticed that the article pages are no longer showing in the Google SERPs (but they are in Bing!) - this was done by using the "site:" search operator.
Have double-checked our robots.txt file too just in case something silly had slipped in, but that's as it should be...
Has anyone else noticed similar behaviour or could suggest things I could check?
Thanks!
-
Well you're on Wordpress and are using YoastSEO. When a Wordpress category is created, a URL is generated for that category.
Your sitemap was created with Yoast:
Sitemap Last Modified |
|
- 2018-08-23 08:10 +01:00
|
- 2018-08-23 08:21 +01:00
|
- 2018-08-23 08:08 +01:00
|
- 2018-08-07 10:40 +01:00
|
- 2018-08-23 08:10 +01:00
|
- 2018-08-23 08:13 +01:00
|
I can see your articles are indexed now, but I would still recommend removing the Wordpress category URL's from your sitemap. Since the sitemap is commonly used for the things you want Google to crawl and index, I would add the article urls and content with "index,follow" webpages directly to your xml sitemap instead of linking the category pages you don't want indexed.
(IE: ie: http://www.genetex.com/sitemap.xml)
Yoast should give you this option in the settings for xml sitemap generation. If not, I would recommend using Screaming Frog to generate the sitemap.
-
Just as an update to this question, I submitted XML sitemaps directly to the blog articles and those pages are still not showing in the Google SERPs. It seems that new pages are discovered quite quickly (as per Google Alerts) but are then dropped from the index within a day or so.
The only pages which are returned consistently are links to the page which allows comments to be added.
The links which were initially identified as broken, were not actually broken so there was nothing to fix there.
Next step I can think of is to attempt some page sculpting by setting a noindex on the comments pages...
If anyone has any more thoughts or ideas, I'd appreciate your input
-
Great - thanks for your help
-
I resubmitted the sitemap for the blog in GWT and no errors were found...
I have to say I am v surprised at the number of dead links - we don't have that many blog posts so unless this is indicating content on our main site (where the pages are still . Even then, as I mentioned to Alan, the only missing content Google Webmaster Tools picks up on is where event tracking is used and it thinks the label is a link.... I did ask Google about these erroneous missing page and they said there was nothing that can be done to indicate they're not meant to be pages and that it would not affect the site's quality.
BTW, An article we published a few hours ago is now showing up in the Google results so it does seem like the rest of the pages have been penalised
Time to figure out what's going on with the missing pages...
Thanks, Irving
-
i sent the list, i had a bit of a look and it may be that they were timing out
-
Thanks Alan, have DM'ed you.
-
Submit a sitemap.xml file for these pages you want indexed, If they are linked to on the site and not blocked in robots.txt they will get indexed again. Definitely fix that sick amount of broken links, Google could be determining that these pages are not worth anything because the links on them are all dead ends.
-
The broken links were found using the Bing api. so bing will see them as such,
If yougive me a email, i willl send you the list
-
39 no-index pages on the blog could be correct with the category pages.
I'm quite surprised at the number of broken links - is this specific to /blog and are they actual links? GWT usually picks up event tracking as broken links...
Good point about the homepage - I should get a canonical tag on that...
Thanks!
-
I found 39 pages that have been no-index, does that add up?
I also found 33,000 broken links.
anouther problem you have is that both http://www.abcam.com/blog/ and http://www.abcam.com/blog/index.cfm are linked to in your site, this means that the pagerank is split. you should link to only http://www.abcam.com/blog/
-
The blog homepage is http://www.abcam.com/blog
@Alan: The rest of the site is indexable, just the the blog area where noindex has been used (the blog homepage and category pages are auto-generated and repeat a lot of the content in the articles)
@Shailendra: Yes, they were indexed - the last Google Alert which specifically highlights content from the blog is mid-June.
-
Firstly, you don't need to write index,follow on normal pages. Secondly, as you say, "no longer showing in Google SERPs", this means that it was earlier indexed, right? Now if it is no longer in Google's index, it means penalization. Please give the url of your website.
-
It may have something to do with the homepage being noindex, as that is unusual.
Can we get a url, I may find what you missed?
-
Hi,
Can you please share URL ?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Disavow links and domain of SPAM links
Hi, I have a big problem. For the past month, my company website has been scrape by hackers. This is how they do it: 1. Hack un-monitored and/or sites that are still using old version of wordpress or other out of the box CMS. 2. Created Spam pages with links to my pages plus plant trojan horse and script to automatically grab resources from my server. Some sites where directly uploaded with pages from my sites. 3. Pages created with title, keywords and description which consists of my company brand name. 4. Using http-referrer to redirect google search results to competitor sites. What I have done currently: 1. Block identified site's IP in my WAF. This prevented those hacked sites to grab resources from my site via scripts. 2. Reach out to webmasters and hosting companies to remove those affected sites. Currently it's not quite effective as many of the sites has no webmaster. Only a few hosting company respond promptly. Some don't even reply after a week. Problem now is: When I realized about this issue, there were already hundreds if not thousands of sites which has been used by the hacker. Literally tens of thousands of sites has been crawled by google and the hacked or scripted pages with my company brand title, keywords, description has already being index by google. Routinely everyday I am removing and disavowing. But it's just so much of them now indexed by Google. Question: 1. What is the best way now moving forward for me to resolve this? 2. Disavow links and domain. Does disavowing a domain = all the links from the same domain are disavow? 3. Can anyone recommend me SEO company which dealt with such issue before and successfully rectified similar issues? Note: SEAGM is company branded keyword 5CGkSYM.png
Technical SEO | | ahming7770 -
No follow links on a blog
Hi On our blog, we have a section called 'Tags'. I have just noticed that these links are all "no follow" links. The tags section does appear on every single page on the blog - is this recommend to have them as 'no follow' links or should I get our developer to change them. Thanks
Technical SEO | | Andy-Halliday0 -
Cached pages still showing on Google
We noticed our QA site showing up on Google so we blocked them in our robot.txt file. We still had an issue with them crawling it so we blocked the site from the public. Now Google is still showing a cached version from the first week in March. Do we just have to wait until they try to re-crawl the site to clear this out or is there a better way to try and get these pages removed from results?
Technical SEO | | aspenchicago0 -
Is it bad to have your pages as .php pages?
Hello everyone, Is it bad to have your website pages indexed as .php? For example, the contact page is site.com/contact.php and not /contact. Does this affect your SEO rankings in any way? Is it better to have your pages without the extension? Also, if I'm working with a news site and the urls are dynamic for every article (ie site.com/articleid=2323.) Should I change all of those dynamic urls to static? Thank You.
Technical SEO | | BruLee0 -
Could a large number of No Followed links in to my site have caused a penalty?
On 22nd Feb, I placed a text ad on a respected industry recruitment website. As the site has many pages, the webmaster made a point of NOFOLLOWING the links back to my site. 6 days later, my site lost 30% of its daily traffic overnight, and it's stayed there ever since. I looked on Webmaster Tools just now, and it says that there are 125,000 links coming in to my site from the site I'm advertising on (even though the links are NOFOLLOWED). Could this have triggered the fall in search traffic to my site? Thanks for any feedback.
Technical SEO | | tofftrader0 -
Would nofollowing the footer throw an unnatural blance between followed and nofollowed links?
I have been getting errors for too many on-page links. All the major navigation pages are found in links within the navigation tabs and are identical to the footer links. So my question is, would nofollowing the footer look unnatural and throw off the balance between followed and nofollowed links on the site and negatively effect SEO?
Technical SEO | | smilingbunny0 -
Page crawling is only seeing a portion of the pages. Any Advice?
last couple of page crawls have returned 14 out of 35 pages. Is there any suggestions I can take.
Technical SEO | | cubetech0 -
Adding no follow links on my site
I am getting a warning about having too many links on my page www.accessoriesonline.co.uk (152) but I don't want to remove any links from the site. Its an ecommerce site with categories across the top, featured products and then a further category navigation in the footer. Would it be beneficial if I added a rel="nofollow" to the links in the footer as these are duplicates of the one's in the header or would this harm the links in the header and the destination URL's which I definitely want to be crawled? Also, does anyone know if SEOMOZ considers links with a rel=nofollow as an actually link when they calculate their overview? Thanks in advance
Technical SEO | | gavinhoman0