Welcome to the Q&A Forum

Thanks, however, the meta tag won't work in this case because it's technically one page with an infinite amount of names via the URL (remember, the CMS only depends on the 123.html and ignores the directories preceding it). If I applied the NOINDEX within the meta, then the version I do want to get indexed would not be indexed.

The question was really around "will the internal rewrite of /123.html to just /123 be impacted if we disallow */123.html" - and since the rewrite happens before the bot sees it, I presume the answer is "no, it will not be impacted: 123.html will be blocked yet /123 will still be indexed.

Now, after I posted the question I realized this is the case where I should use a "greedy" 301 redirect via htaccess rather than try to block permutations of the URL via robots.txt. So I decided to not go the robots.txt route and instead do a 301 redirect via regex:

*/123.html to /123 (that's obviously not perfect regex, but you see my point)

Our not-so-lovely CMS loves to render pages regardless of the URL structure, just as long as the page name itself is correct. For example, it will render the following as the same page:

example.com/123.html

example.com/dumb/123.html

example.com/really/dumb/duplicative/URL/123.html

To help combat this, we are creating mod rewrites with friendly urls, so all of the above would simply render as example.com/123

I understand robots.txt respects the wildcard (*), so I was considering adding this to our robots.txt:

Disallow: */123.html

If I move forward, will this block all of the potential permutations of the directories preceding 123.html yet not block our friendly example.com/123?

Oh, and yes, we do use the canonical tag religiously - we're just mucking with the robots.txt as an added safety net.

I may be going against the grain here, but I'm going to throw this out there and I'm interested in hearing your feedback...

We are a fairly large online retailer (50k+ SKUs) where all of our category and subcategory pages show well over 100 links (just the refinement links on the left can quickly add up to 50+). What's worse is when you hover on our global navigation, you see the hover menu (bot sees them as

• ) of over 80 links.

  Now I realize the good rule of thumb is not to exceed 100 links on a page (and if you did your math, you can see we already exceeded that well before we let the bots get to the good stuff we really wanted them to crawl in the first place).

  So...

  Is it wise to intentionally shield these global nav links from the bots by using javascript?