Welcome to the Q&A Forum

Definitely review George's comment as you need to figure out why they're being crawled. As Andrea said, any solution takes time, I'm sorry to say. Robots.txt is not a good solution for getting pages removed that are already indexed, especially in bulk. It's better at prevention than cure.

META NOINDEX can be effective, or you could rel=canonical these pages to the appropriate non-AJAX URL - not sure exactly how the structure is set up. Those are probably the two fastest and more powerful approaches. Google parameter handling (in Webmaster Tools) is another option, but it's a bit unpredictable whether they honor it and how quickly.

You can only do mass removal if everything is in a folder, if I recall. There's no way to bulk remove unless all of the pages are structurally under one root URL.

Can you private message me (via the site) and tell me the URL - I'll check the campaign and ask the team, in case we've got a bug.

It does, although Google seems to be slightly less fond of it over time. Since I wrote my reply in March, rel=prev/next are actually beginning to be more effective. I've never seen any major issues with NOINDEX'ing pages 2+, though. In many cases, it's just a lot easier to implement.

The big issue this year is that Google sometimes just ignores deindexation signals. So, you really have to try it and see.

I'd also add that I'm talking about search pagination here, not article pagination. Rel=prev/next is a much better choice for article pagination, because the content is unique across pages. Indexing page 11 of search results isn't much of a benefit, in most cases.

I just want to add that the answer does depend a lot on the content. If it's just an issue of duplicate title and META tags, then any uniqueness does help. Generally, though, use the actual product names and information - you can easily generate the tags from the database for most data-driven sites.

If it's an issue of duplicate URLs for the same products, that's a different matter. In other words, make sure that your navigation (searches, sorts, paginations) isn't spinning out copies of your product pages. If it is, then you'll want to control those duplicates and canonicalize or de-index them. If you have 1000s of duplicate product titles and only 200 products, then you're creating duplicate product pages. If you have 1000 duplicate titles and 1000 product pages, then you just need to create unique tags.

GIven that you're talking about 5-6 pages, I'd tend to agree with Davinia. It sounds like these pages have a shot at having unique value, and you could probably beef them up with solid content relatively easily. If you were spinning out 100s or 1000s of thin, geo-targeted pages, that would be different (and that can cause Panda problems), but for half a dozen pages, the canonical would have very little impact here. If they're valuable to users, let them work for you and make them valuable to search.

Are these pages identical, but just reached via different paths (based on category/sub-category navigation)? If so, the canonical tag is definitely a good choice here. Ideally, it's best not to create duplicate paths, as it can cause long-term problems, but in lieu of that, canonical is probably your best defense.

It's fine to not include the canonical on the main/target page. As @donford said, it's fine to have a self-reference canonical, but it's definitely not necessary.

Are these live pages, or did something page (I can't seem to access them). Looking at the URLs, it seems like these are slightly different products, so I'm not clear if the canonical tag is appropriate.

I think this is borderline and the kind of thing Panda hit in some cases. While the pages aren't true duplicates, they do appear a bit thin - they're basically the same content with a few brand-related keywords swapped out. It really comes down to scope, though - if you spun out 100s or 1000s of these, you could really dilute your index and even creep into potential Panda penalty territory.

With about 600 pages in the index, though, and these accounting for only 19 of them, it seems like your risk is pretty low. It would be great to build out the content and make it more unique, but I don't think you're in any immediate danger. You don't want to NOINDEX or block these pages somehow, because I assume you're trying to get some mileage out of the long-tail keywords. My best advice is to try to beef them up and don't spin out too many variants (100+ of these pages with a 600 page site would be dangerous).

Irving's correct - that tag is basically nonsense, as best I can tell. It's just a comment. There's no approved tag to block part of a page from Google. Google has a directive that looks similar to this, but it's for Google Search Appliance (internal site search) and doesn't impact organic search (we actually just tested it recently, coincidentally).

I disagree a little bit about cross-domain canonical. I have some friends at major media properties who have had good luck using cross-domain canonical for syndicated content, and it does seem to pass link-juice when used appropriately. It is true that Google doesn't always honor the cross-domain use, so it's very situational.

301-redirects are certainly more reliable, but they are different in the sense that a canonical will only remove the source page from search, whereas a 301 will remove it for users as well. So, it really depends on what you're trying to accomplish.

So, you basically (in some cases) have a canonical pointing to a URL, and then that URL 301s to a new URL? If that's the case, I think Nakul is right - it's not ideal. It's not a disaster, but you're basically putting Google through extra hops and you could be losing link-juice. Unfortunately, it's nearly impossible to calculate how much. Ideally, I'd change the canonicals to point to the new URLs.

I think Screaming Frog is just warning you that the canonical version doesn't seem to match the display URL. They can't really tell (we have the same problem in SEOmoz tools) what the "right" canonical is - they can just warn of a mismatch.

I'm a bit confused as to the purpose of the dual URLs here. The best canonical implementation is to use one URL. The canonical tag can act as a band-aid, but  consistency is still the best defense. Having multiple paths to the same page is rarely beneficial.

Is this a CMS, or are you trying to make the canonical only impact certain pages or conditions? The canonical tag is just an HTML tag, in the header. It doesn't really need special code, unless the target of the canonical has to be created dynamically. Unfortunately, then the solution depends entirely on your implementation (it's not one-sized-fits-all across all PHP sites).

If you could describe the structure of the site a bit (example URLs and how they're created) without revealing the site, that could help us pin it down.

It's tricky for home pages. Google really tends to prefer the root URL, and it's best if you can target that. Unfortunately, .Net has a bad way of forcing you to use a deeper page.

If you can't resolve to the root, then you need to be consistent with your internal links before you set the canonical. If you're resolving and linking internally to:

/keywords/default.aspx

...then canonical to that page. It's not quite as good as the root, but by setting the canonical to "/keywords" you could actually be creating a third URL that isn't represented in either your inbound or internal links.

In other words, the first step to a good canonical implementation is to actual use ONE URL, no matter what it is. The canonical tag itself is a bit of a band-aid. It's effective, but fixing the on-page structure is the first, best step.

There's no ideal solution here, but I think you may have to try it, just to focus the ranking power and try to sort out what's going on.

Single-page penalties are uncommon but possible. Is the page indexed? That's step one. If it is, are you seeing duplicates when you search your index (use the "site:" operator) for the page's title? If so, it could be a filter. If you're indexed but not ranking even on exact-match terms (in quotes), it could be a penalty.

Is that the source or target of the redirect? When I check the header on that page, I'm seeing a regular 200 (OK).

Since you use the non-www version of the URLs, I tried them with "www.", but those seem to be 301-redirecting properly. I'm not seeing any 302s offhand.

I think that's probably fine - we're warning you just to tell you the pages are blocked, but blocking them sounds perfectly sensible. Is Google indexing them at all?

I assume that the login page is getting multiple URLs, possibly based on the path people take to it? Ideally, it would help to collapse that to one URL (store the parameters in a form field or visitor session/cookie), but my gut reaction is that that's a nice to have. I suspect you're probably ok.

Ah... yeah, that's tricky. There's no magic solution, I'm afraid. You've really got three options:

(1) Leave it alone

(2) Re-organize your site architecture to push individual date pages down a level or two, so that they get less internal link-juice.

(3) Re-organize such that you focus search engines on chunks of time or maybe date/aspect combinations, but then de-index the individual date combos. This would take a much better understanding of your site structure than I currently have. The goal would be to focus your index on some smaller combination of pages that still covers 80% of your search traffic.

The big problem is just that this is a lot potential dilution, and I suspect that many of these pages look very similar to Google. I'm also certain that not all pages have the same value, either for SEO or users, so there's some hybrid approach where you could prune back but not lose everything. Long-term, I think that's worth the time and trouble to sort out, but it's not an emergency or something I'd rush into.

Google is seeing the same Robots.txt content (in GWT) that you show in the physical file, right? I just want to make sure that, when the site was hacked, no changes were made that are showing different versions of files to Google. It sounds like that's not the case here, but it definitely can happen.

The 301 should override the canonical and hreflang, but if it were me I'd at least remove the canonical - I'm not a fan of mixed signals.

That's odd: "008" appears to be the user agent for "80legs", a custom crawler platform. I'm seeing it in other Robots.txt files.

Not honestly sure what User-agent "008" is, but that seems harmless. Why the crawl delay? There are better ways to handle that than Robots.txt, if a crawler is giving you trouble.

Was there a specific message/error in GWT?

The NOINDEX will take some time to remove and will kill your link equity (for now) - that's my main hesitation. I get why you don't want to 301, for user reasons. The canonical will still carry link-juice over. Personally, I think I'd give the canonical time - your current situation isn't ideal, but the NOINDEX could waste your inbound links and, now that you've canonical'ed, even harm the .co.uk site slightly. Better to give your .ie site visitors a path to the .co.uk site/prices and live with some minor Google hiccups, IMO.

The other simple reality is that, if Google isn't honoring the canonicals, they may ignore the NOINDEX tags or take a very long time to process them. The most definitive solution would be removal in Google Webmaster Tools, but that's pretty extreme.

I'd tend to agree with Troy (and I'm thumbing up his replies). This isn't your home-page or sitewide navigation - having too many links on just this one page probably isn't worth worrying about too much. You could nofollow the tag links - they are redundant. Realistically, though, since they link to the same URLs, Google probably already ignores those links (even though we count them twice).

You could argue that maybe the page isn't that helpful for users and prune it down somehow, but purely from an SEO standpoint, it's not a big deal. The main issue with too many links is one of dilution. I wrote about it more here:

http://www.seomoz.org/blog/how-many-links-is-too-many

Yeah, it seems like Google is overriding the canonical, just based on the Google.ie connection, but it's really hard to tell. I'm actually conversing with some other SEOs about this same problem and Google's mixed signals on hreflang vs. canonical (or both together), and the answers aren't very clear. Different Google reps have given slightly different suggestions, and none of them are working consistently in all cases.

If you're going to drop the .ie website temporarily, I'd probably 301-redirect it. It's a little difficult to reverse, but at least you'll consolidate all of your link-juice and ranking factors for that site into the .co.uk site. If you just deindex the Irish domain, you'll lose what SEO value you've built to it.