Welcome to the Q&A Forum

Unfortunately, I don't think there are many reliable options, in the sense that Google will always honor them. I don't think they gauge crawl frequency by the "expires" field - or, at least, it carries very little weight. As John and Rob mentioned, you can set the "changefreq" in the XML sitemap, but again, that's just a hint to Google. They seem to frequently ignore it.

If it's really critical, a 304 probably is a stronger signal, but I suspect even that's hit or miss. I've never seen a site implement it on a large scale (100s or 1000s of pages), so I can't speak to that.

Two broader questions/comments:

(1) If you currently list all of these pages in your XML sitemap, consider taking them out. The XML sitemap doesn't have to contain every page on your site, and in many cases, I think it shouldn't. If you list these pages, you're basically telling Google to re-crawl them (regardless of the changefreq setting).

(2) You may have overly complex crawl paths. In other words, it may not be the quantity of pages that's at issue, but how Google accesses those pages. They could be getting stuck in a loop, etc. It's going to take some research on a large site, but it'd be worth running a desktop crawler like Xenu or Screaming Frog. This could represent a site architecture problem (from an SEO standpoint).

(3) Should all of these pages even be indexed at all, especially as time passes? More and more (especially post-Panda), more indexed pages is often worse. If Googlebot is really hitting you that hard, it might be time to canonicalize some older content or 301-redirect it to newer, more relevant content. If it's not active at all, you could even NOINDEX or 404 it.

I think John is right on all counts. I'll just add that, while the canonical tag should work fine, you may want to consider whether it makes sense to index the premium content at all. It really boils down to whether you think people will link to the premium versions, in which case the canonical tag would help consolidate that link-juice.

The other option, of course, would be to roll-out the wrappers/upgrades dynamically and not even have two copies - everyone would see the same URL, but premium members would see it differently. In most cases, I think that would be the best bet for SEO. It also means you're not maintaining two copies (although I'm guess your current set-up is at least partially dynamic).

Adding to Daniel's comment, I'd say the big difference "...through our faceted search." It's important to have both the XML entries and a crawl path. An XML sitemap may be enough to get the pages indexed, but they won't inherit any internal link-juice. That comes through your internal links. Somewhere, there needs to be a link that Google can crawl to the other cities.

The direct back-links will help, and should get you indexed and possibly ranking, but you're still losing the authority from the domain as a whole that you'd inherit via internal links. The upshot is that you'll lose ranking power.

Oh, sorry - I thought the blog was on a separate domain and you were just setting the reverse proxy to change it from "blog.domain.com" to "domain.com/blog". In this case "www.mydomainname.com" is actually the main site, right?

Done right, that should work - as long as Google sees the blog as a sub-folder of the main domain, the physical separation across servers shouldn't matter much. People separate out servers for all kinds of legitimate reasons.

I guess it depends on why this is on a separate domain at all. If you have the technical capacity to host the blog content under your company website, then in 95%+ of cases, I'd move the blog. Unless there's a clear need to brand a separate domain or legally separate the blog from the company website, having a separate domain is just splitting your authority. All the links that come to your blog are going to just funnel into 1 linking domain to your main site. If that content was on your main site, you'd have dozens or (down the road) hundreds of linking domains instead.

If it has to be separate, than I agree with Andrea - I wouldn't copy it. You're setting up duplicates and confusing your potential audience. In general, though, I'd want to understand why you made the split. Fully integrating them has much more bang for your buck, SEO-wise.

If they're really just low quality (spun articles, blog comments, etc.), then I agree with Casey. You can cause more trouble taking a machete to your links - move forward with stronger, better links, and the past will probably be forgiven. My only caveat is that if these links appear to be paid or farmed, I might be a bit more aggressive with removing some of them.

They shouldn't - a META NOINDEX is easier to undo than a Robots.txt block, 301, or canonical tag, in my experience. The biggest risk is just a delay - it may take Google a little time to re-index the content once you remove the tag.

What I wouldn't do is add/remove the tag rapidly. For example, if you had a product that went out of stock every other day, I'd leave it alone - Google wouldn't respond quickly enough to all those changes. So, once a category has enough results, I'd lift the NOINDEX permanently. It's really just a move to consolidate while you build up the site - both in terms of content and your link profile.

Unfortunately, the painful reality, especially if you've been hit by Panda, is that you probably can't support that scale or that it looks thin to Google. 500 cities X 50 categories = 25,000 "category" pages, so to speak, all of which are basically just search results. For most sites, it's just too much.

I'd definitely keep the cities as sub-folders. If you go the sub-domain route, you could fracture your internal link-juice even more. It depends a bit on the authority and marketing budget of the site. If each city is a separate property with its own sales force, budget, etc., there may be a logic to sub-domains. Unless you're Groupon or someone like that, though, it's probably a bad idea.

You may have to prune down the indexed content, to be frank. I'd look for other Panda factors, too, like aggressive ad density (too many ads to too little content) or very thin pages. If you have tons of cities or categories with no listings, META NOINDEX them. You could even do it dynamically - only let Google index a page if it has 1+ listings, for example.

I'd also take a look at other low-value content, like paginated search. If each city has 100s of pages and you're indexing page 2, page 3, etc., consider consolidating them. It's a tricky topic, but Adam Audette has a great write-up here:

http://searchengineland.com/five-step-strategy-for-solving-seo-pagination-problems-95494

These pages can look very low-value to Google. Add in search sorts and other variants, and your 25K categories could be exploding into hundreds of thousands of pages, before Google even gets to the listings themselves. The ads are the real meat of the site, and that's where you want Google to focus.

I wish I could convince people that more DOES NOT EQUAL better when it comes to index size. You'd think Panda would've been the nail in that coffin, but too many webmasters are still operating in 2005.

I've never seen an issue where a large-scale META NOINDEX caused Google to get suspicious. It's possible to NOINDEX the wrong pages and lose traffic, but Google generally doesn't get jumpy about it like they would a large scale 301-redirect (where you might be PR-sculpting).

If these are really duplicates, canonical tags might be a better bet. Honestly, while I agree with Stephen 99.9%, if there's no glaring current issue, you could ease into it. Start with the worst culprits - obvious, 100% duplicates. That should be an easier sell, too. If you can't sell the larger cut, it's not going to matter.

Agreed with Alan (deeper in the comments) - you may have cut off links to these pages or internal link-juice flow. It would be much better to either 301-redirect the "/shop" pages or use the canonical tag on those pages. In Apache, the 301 is going to be a lot easier - if "/shop/product" always goes to "/product" you can set up a rewrite rule in .htaccess and you don't even need to modify the site code (which site-wide canonical tags would require).

The minor loss from the 301s should be much less than the problems that may have been created with Robots.txt. As Alan said, definitely re-point your internal links to the canonical (non-shop) version.

Haha - love it Thanks for the extra dose of holiday spirit.