I'd have to disagree with David, strongly so.

But then it is probably a matter of what "tag page" means for you and what you do with your tag pages!

Is a tag just something you slap on articles and stuff to simply have all a page that merely displays them all? Well yeah, then I'd tend towards noindex. I'd also wonder how many people really use a tagging like that - and have a use for it.

Personally I favour the other approach where tag pages are made into a page that can stand alone, for example giving a general overview over a topic and then listing the tagged articles for more specialised content. (Yes, you need a certain minimum of articles for tags to be useful at all ... and too many of one tag is probably also a warning sign of over-specialisation.) Of course this is not possible with every website and organisation. I have found, though, that such "hub pages" work exceptionally well for more generic than long term searches.

TL;DR: The importance and treatment of tags and tag pages largely depends on how you define their role.

Nico

"100 link rule": At some point in early 2016, John Müller or sb. else at Google said "a reasonable number / a few thousand links at most". Unfortunately I did only save the statement, not the exact source/date.

Nico

I don' think Martijn's statement is quite correct as I have made different experiences in an accidental experiment. Crawling is not the same as indexing. Google will put pages it cannot crawl into the index ... and they will stay there unless removed somehow. They will probably only show up for specific searches, though

In September 2015 I catapulted a website from ~3.000 to 130.000 indexed pages (roughly). 127.000 were essentially canonicalised duplicates (yes, it did make sense) but also blocked by robots.txt - but put into the index nonetheless. The problem was a dynamically generated parameter, always different, always blocked by robots.

The title was equal to the link text; the description became "A description for this result is not available because of this site's robots.txt – learn more." (If Google cannot crawl a URL  Google will usually take titles from links pointing to that URL). No sign of disappearing. In fact, Google was happy to add more and more to its index ...

At the start of December 2015 I removed the robots.txt block - Google could now read the canonicals or noindex on the URLs ... the pages only began dropping out, slowly and in bunches of a few thousand in March 2016 - probably due to the very low relevancy and crawl budget assigned to them. Right now there are still about 24.000 pages in the index.

So my answer would be: No - disabling crawling in the robots.txt will NOT remove a page from the index. For that you need to noindex them (which sometimes also works if done in robots.txt, I've heard). Disallowing URLs in the robots.txt will very likely drop pages to the end of useful results, though, as Andy described. (I don't know if this has any influence on the general evaluation of the site as a whole; I'd guess not.)

Regards

Nico

There has even been a Google Webmaster Guidelines Update in February 2016 which states

"Limit the number of links on a page to a reasonable number (a few thousand at most)."  (Source: https://support.google.com/webmasters/answer/35769?hl=en)

So I really would not bother too much, especially not in a navigation - it often makes lots of sense to have lots of links there. (For example I have several alphabetical selections available on hover from main categories. It would not make sense to remove them just to have fewer links.)

More links are of course not always better - consider what the user is likely to expect/need in navigation etc. Of course, more links mean that the relative importance of each link decreases; but google is able to identify navigation and repeating elements that appear on every page. I'd assume that they treat them different to main-content links. Because, well, they feel a lot different.

Regards

Nico

In the revised webmaster guidelines, google says "[...] Provide a sitemap file with links that point to the important pages on your site. Also provide a page with a human-readable list of links to these pages (sometimes called a site index or site map page)." (Source: https://support.google.com/webmasters/answer/35769?hl=en)

I guess what they mean by this is something like this: http://www.ziolko.de/sitemap.html

Still, I wonder why they say that. Just to ensure that every page on a site is linked and consequently findable by humans (and crawlers - but isn't the XML sitemap for those and gives even better information)? Should not a good navigation already lead to every page? What is the benefit of a link-list-page, assuming you have an XML sitemap? For a big site, a link-list is bound to look somewhat cluttered and its usefulness is outclassed by a good navigation, which I assume as a given. Or isn't it?

TL;DR: Can anybody tell me what exactly is the benefit of a human-readable list of all links?

Regards,

Nico

My take: Do not use bold/italics/underline to mark keywords; use them to mark up important parts in your text in a consistent way. That can be the same but needn't be. For example, I tend to italicise proper Names and quotes; I tend to bold the most important points of a text so somebody can skim the page, immediately see the most important points and decide to read the rest or not (in addition to subheadings); I outright avoid underlines. Those are for links.

Nothing of that has to do with search engines. Its aim is improved legibility/"scannability". I doubt google would directly penalise for marking keywords up like this - it might worsen the user experience, though.

Nico

Thanks - so I have to continue the search for where a tenfold increase in indexed pages (according to Search Console) might possibly come from. Sadly, the rest of your reply misses my problem; probably I have been unclear.

The reason I was asking for a method to know what pages ARE indexed is: I seem to have no problem getting stuff indexed (crystal-clear sitemap with dates; clear link structure &c.) but google seems over-eager and indexes more than there really is. If it is some technical problem, I'd like to fix that - but Google does not show anywhere what pages are actually indexed. There are lots of methods around - but none that I found do work as of now.

I have been well aware of JumpTo-Links, as I stated, and it works nicely. No problem at all with "not enough" indexed pages - really rather the opposite with no idea what causes it.

Regards

Nico

Hi,

Back in September, I added a function that sets an anchor on each subheading (h[2-6]) and creates a Table of content that links to each of those anchors. These anchors did show up in the SERPs as JumpTo Links. Fine.

Back then I also changed the canonicals to a slightly different structur and meanwhile there was some massive increase in the number of indexed pages - WAY over the top - which has since been fixed by removing (410) a complete section of the site. However ... there are still ~34.000 pages indexed to what really are more like 4.000 plus (all properly canonicalised). Naturally I am wondering, what google thinks it is indexing. The number is just way of and quite inexplainable.

So I was wondering:

Does Google save JumpTo links as unique pages?

Also, does anybody know any method of actually getting all the pages in the google index? (Not actually existing sites via Screaming Frog etc, but actual pages in the index - all methods I found sadly do not work.)

Finally: Does somebody have any other explanation for the incongruency in indexed vs. actual pages?

Thanks for your replies!

Nico

I think that page anchor's importance is increasing. Moz has pointed out that long content tends to rank better (for in-depth articles) than short content. The (shortened) logic being that it is better to present one articles that gives readers everything (and maybe a bit extra) rather than forcing them to do another search.

Personally I am a fan of a Table of Contents that links to the sub-headings in an article. (For Wordpress there is the rather good plugin Table of Contents Plus. It ads an anchor to all <h[1-6]>with an id that equates to the heading, replacing certain invalid signs.)  Google also likes this and USES the anchors: When you Google for https://www.google.com/search?q=wikipedia+en+SEO+Methods&ie=utf-8&oe=utf-8 , for example, you will get an extra "Jump to"-link that gets you directly to the relevant paragraph (or what google thinks is the most relevant paragraph for your search query anyway). This, of course, needs anchors.</h[1-6]>

I do not know how many people actually use these anchor-links and would be very much interested in that data. Personally I find this very nifty and would recommend it. It is not that much extra effort in most cases and it adds a very useful feature - also for linkbuildling and sharing.

You should of course consider possible implications for tracked scroll-depth and such stuff!

And for exactly that reason I would here chose different pages for tables, chairs etc - how would you know what people were searching for exactly? (So far I mostly used anchors for articles, not for different product sections). As I see it, that starting page is more a "furniture" landing page than a dedicated "tables" or "chairs" landing page.

From what I observed, it seems to be the datePublished. Example is this page http://www.grimoires.de/buch/1/. You see "Diese Rezension wurde veröffentlicht am 31.12.2002 und zuletzt geändert am 16.05.2015." (with appropriate schema.org data) below the writer info in the main part. This info is not cheated or manipulated in any way, also used in the sitemap and accurate (it takes the unix time stamp of the time of creation/modification). In a SERP, 31.12.2002 appears. Same for another page that had the content (text) extended.

What might influence it: changes in 2015 were rather minimal - the meta description only, if I remember correctly. I am not sure, if the extent of a change might push google towards displaying the dateModified.  It would seem reasonable to me to expect a "change threshold" but in the end I simply don't know. Anybody with hard empiric data?

Regards,

Nico

My take: Do not use bold/italics/underline to mark keywords; use them to mark up important parts in your text in a consistent way. That can be the same but needn't be. For example, I tend to italicise proper Names and quotes; I tend to bold the most important points of a text so somebody can skim the page, immediately see the most important points and decide to read the rest or not (in addition to subheadings); I outright avoid underlines. Those are for links.

Nothing of that has to do with search engines. Its aim is improved legibility/"scannability". I doubt google would directly penalise for marking keywords up like this - it might worsen the user experience, though.

Nico

I recently posed this question the other way around, i.e. a site had different pages (technically totally unconnected) for each product variant and it caused certain problems: https://moz.com/community/q/duplicate-content-through-product-variants

I am not sure if this 100% applies to you: I would not use "different model" and "variant" as synonyms. If they are different models, they are different, unique products and have distinct features, advantages, maybe extras etc. Variants, to me, are for example different colour, maybe the same model with different extras or similar. There might be some grey zone where different models might actually be quite like variants, differing little from each other - then I'd ask if they differ enough to say something about each and/or contrast them.

For different models I would in fact chose different pages that go deeper into details. (Think of different books from the same publisher!) For variants, having more than one page is problematic. If I had the free choice, I'd bundle all variants on a single page with dropdowns to select the variant. There is also the question if variants are sufficiently different to write unique content for each - if that is the case, it could justify separate pages. A definite benefit of those is that they can be linked directly.

Regards,

Nico

I think that page anchor's importance is increasing. Moz has pointed out that long content tends to rank better (for in-depth articles) than short content. The (shortened) logic being that it is better to present one articles that gives readers everything (and maybe a bit extra) rather than forcing them to do another search.

Personally I am a fan of a Table of Contents that links to the sub-headings in an article. (For Wordpress there is the rather good plugin Table of Contents Plus. It ads an anchor to all <h[1-6]>with an id that equates to the heading, replacing certain invalid signs.)  Google also likes this and USES the anchors: When you Google for https://www.google.com/search?q=wikipedia+en+SEO+Methods&ie=utf-8&oe=utf-8 , for example, you will get an extra "Jump to"-link that gets you directly to the relevant paragraph (or what google thinks is the most relevant paragraph for your search query anyway). This, of course, needs anchors.</h[1-6]>

I do not know how many people actually use these anchor-links and would be very much interested in that data. Personally I find this very nifty and would recommend it. It is not that much extra effort in most cases and it adds a very useful feature - also for linkbuildling and sharing.

You should of course consider possible implications for tracked scroll-depth and such stuff!

And for exactly that reason I would here chose different pages for tables, chairs etc - how would you know what people were searching for exactly? (So far I mostly used anchors for articles, not for different product sections). As I see it, that starting page is more a "furniture" landing page than a dedicated "tables" or "chairs" landing page.

There has even been a Google Webmaster Guidelines Update in February 2016 which states

"Limit the number of links on a page to a reasonable number (a few thousand at most)."  (Source: https://support.google.com/webmasters/answer/35769?hl=en)

So I really would not bother too much, especially not in a navigation - it often makes lots of sense to have lots of links there. (For example I have several alphabetical selections available on hover from main categories. It would not make sense to remove them just to have fewer links.)

More links are of course not always better - consider what the user is likely to expect/need in navigation etc. Of course, more links mean that the relative importance of each link decreases; but google is able to identify navigation and repeating elements that appear on every page. I'd assume that they treat them different to main-content links. Because, well, they feel a lot different.

Regards

Nico

I don' think Martijn's statement is quite correct as I have made different experiences in an accidental experiment. Crawling is not the same as indexing. Google will put pages it cannot crawl into the index ... and they will stay there unless removed somehow. They will probably only show up for specific searches, though

In September 2015 I catapulted a website from ~3.000 to 130.000 indexed pages (roughly). 127.000 were essentially canonicalised duplicates (yes, it did make sense) but also blocked by robots.txt - but put into the index nonetheless. The problem was a dynamically generated parameter, always different, always blocked by robots.

The title was equal to the link text; the description became "A description for this result is not available because of this site's robots.txt – learn more." (If Google cannot crawl a URL  Google will usually take titles from links pointing to that URL). No sign of disappearing. In fact, Google was happy to add more and more to its index ...

At the start of December 2015 I removed the robots.txt block - Google could now read the canonicals or noindex on the URLs ... the pages only began dropping out, slowly and in bunches of a few thousand in March 2016 - probably due to the very low relevancy and crawl budget assigned to them. Right now there are still about 24.000 pages in the index.

So my answer would be: No - disabling crawling in the robots.txt will NOT remove a page from the index. For that you need to noindex them (which sometimes also works if done in robots.txt, I've heard). Disallowing URLs in the robots.txt will very likely drop pages to the end of useful results, though, as Andy described. (I don't know if this has any influence on the general evaluation of the site as a whole; I'd guess not.)

Regards

Nico