It is correct that DA, PA, depth of pages, etc. are all factors in determining which pages get indexed. If your site offers good navigation, reasonable backlinks, anchor text, etc then you can get close to all pages indexed even on a very large site.

Your site map should naturally include a date on every link which indicates when content was added or changed. Even if you submit a 10k list of links, Google can evaluate the dates on each link and determine which content has been added or modified since your site was last crawled.

what could cause one page to drop so much so quickly whilst other pages improved their rank?

Without knowing the page URL and keyword, we are left to guesswork. As Sean suggested, the most three likely issues are either:

1. Google has made an algorithmic adjustment which was unfavorable to that page.

2. That page has been penalized in some form. It could be there is an issue with the page's content, or perhaps a link to the page has issues and has been devalued.

3. There is a crawl or other issue preventing page rank from flowing properly to that page.

Your sitemap should include every page of your site that you wish to be indexed.

The idea is that if your site does not provide crawlable navigation, Google can use your sitemap to crawl your site. There are some sites that use flash and when a crawler lands on a page there is absolutely no where for the crawler to go.

If your site navigation is solid then a sitemap doesn't offer any value to Google other then an indicator of when content is updated or added.

Very nice addition John.

How does a canonical work? Does the robot read the canonical and immediately go to the canonical URL or does it continue to read past the canonical tag and get to the no index, follow tag if there is one present?

The first thing to understand is the canonical tag is a suggestion, not an order. While a search engine will usually honor the canonical tag, there are instances where Google or other SEs may determine the canonical tag is not being used correctly so they disregard the canonical tag. Based on this understanding, yes the robot will read the entire page regardless of the canonical tag status.

Is it necessary to have both a canonical tag and no index, follow tag in place? Or should the canonical tag be sufficient to avoid duplicate content?

The two tags you mention conflict. You would never use both tags on the same page.

Noindex means you do not wish the page to appear in the search index. The canonical tag means you do wish the content to be included in the search index, but use the canonical URL in the index.

if both a canonical tag and no index, follow tag are in place, should they be in a specific order?

The order of meta tags does not matter. If a page was marked with both a canonical tag and a noindex tag, the noindex tag would take effect and the page would not be indexed, so the canonical tag would not have any effect.

In short, you want to use the canonical tag to resolve duplicate content issues, not the noindex tag.

Due to the penalties we have been considering moving everything under one umbrella and manage local sites in directories e.g. .com/es/keyword1 .com/de/keyword2 - however until the penalties hit the url approach has worked very well for us. Any thoughts?

Generally speaking, you will achieve the best results by consolidating your sites under one domain with a dedicated folder for each country as you described. I would recommend delaying the move until you are sure your sites are not under any penalty.

The advantage you will receive with a single root domain is the consolidation of your Domain Authority. It sounds like your sites were doing well before the penalty. The higher DA can help even further.

The internationalized sites are each hosted with a different root domain keyword1.es keyword2.de - are you still confirming that this should not be causing duplicate content penalties?

Correct, as long as the sites are properly set up to target their target countries. Sites which are dedicated to a specific locale and language would not normally compete in SERPs with other sites that offer similar content in another country and language.

Does your company have that experience and do you provide such services?

While I appreciate the inquiry, my resources have been already dedicated for the remainder of this month. You could take a look at the SEOmoz directory. Please note that anyone can list their company in the directory. A listing is not an endorsement.

If you desire a further recommendation you can send me a message on SEOmoz and I will respond. I can share a few names of SEOs whom I have confidence in based on their Q&A responses, blogs and reputation if that would be helpful.

Is there something like an IP specific Google penalty that can apply to web properties across an IP or can we assume Google just picked all pages registered at Google Webmaster?

Think of Google as an intelligent business. They have processes which algorithmically penalize websites. They also have systems which flag sites for manual review. When a penalty is deemed appropriate it is possible for it to be applied on any number of factors such as an IP address, a Google account, a domain, etc. It depends on how widespread of a violation has occurred.

What is the most likely cause for our penalty given the background information? Given the drops started already in November 2010 we doubt that the Panda updates had any correlation t this issue?

You mentioned a few points which can potentially lead to a penalty. I am not clear from your post, but is sounds like you may be linking to casino and gambling sites. While those sites may be legitimate, many have a reputation for using black hat SEO techniques.

If you want to remove a penalty, be certain that you do not provide a followed link to any questionable site. When you provide a followed link to a site, you are basically saying "I trust this site. It is a good site and I endorse it". If you are found to offer a link to a "bad" site, your site can be penalized.

What are the best ways to resolve our issues at this point? We have significant history data available such as tracking records etc. Our actions so far were reducing external links, on page links, and C-class internal links

Hire a professional SEO to review your site. You want to review every page to ensure your site is within Google's guidelines. I am highly concerned about your site's links to external sites. I am also concerned about the automated link building that your current SEO has been doing. A professional SEO company should not lead your site to incur a penalty. I am having difficulty understanding how this happened in the first place, how it has not been fixed in almost a year, and how this SEO company is building links for you. Frankly, it's time to consider a new SEO company.

Translating content to other languages is fine. You can take the exact same article and offer a translated version for each language, and even country. For example you can offer a Spanish version for your Spain site, and a different Spanish version for your Mexico site. As long as these sites are targeting specific countries then there is no duplicate content issues.

After all this time wo/ resolution, should we be moving on two new domains and forwarding all content as 301s to the new pages? Are the things we need to try first?

The penalty would follow to your new domain.

The external linking structure of the pages is very keyword and main-page focused, i.e. 90% of the external links link to the front page with one particular keyword

Not good at all.

Summary: your site needs careful, professional review by a SEO professional who adheres to white hat techniques. Every day your site is penalized you are losing traffic and money. The cost you pay to fix this issue may be extremely small in comparison to the amount of revenue you have lost.