Welcome to the Q&A Forum

As part of a Panda recovery initiative we are trying to get an as comprehensive list of currently URLs indexed by Google as possible. Using the site:domain.com operator Google displays that approximately 21k pages are indexed. Scraping the results however ends after the listing of 240 links. Are there any other sources we could be using to make the list more comprehensive? To be clear, we are not looking for external crawlers like the SEOmoz crawl tool but sources that would be confidently allow us to determine a list of URLs currently hold in the Google index. Thank you /Thomas

An international portfolio of web sites has been suffering from Panda and Penguin over the last months. Essentially many of the international sites completely disappeared from the top 50 when before we had many Top 5 positions for competitive key words. Today we noticed that it appears that for all this time rankings for local keywords have not been affected for domains outside of the most relevant Google CCTLD for the page. To give an example: Domain.it / Keyword 1 (IT) = Italian Competitive Keyword Google.it: Keyword 1 (IT) April 2012 Position 4
Google.it: Keyword 1 (IT) December 2012 Position --- Google.com: Keyword 1 (IT) April 2012 Position 1
Google.com: Keyword 1 (IT) December 2012 Position 1 Google.de: Keyword 1 (IT) April 2012 Position 3
Google.de: Keyword 1 (IT) December 2012 Position 4 Have other people observed such behavior? Does this give any pointers towards how the recovery strategies should be drafted? We have experienced that the more search volume a keywords has received the harder the Panda/Penguin impact. So one hypothesis could be that because the international Google domains do not receive overly significant traffic for the localized keywords Panda/Penguin protection algorithms are not being applied with the same force. Any thoughts are welcome. /Thomas

We have been hit significantly with Panda and assume that our large index with some pages holding thin/duplicate content being the reason. We have reduced our index size by 95% and have done significant content development on the remaining 5% pages. For the old, removed pages, we have installed 410 responses (Page does not exist any longer) and made sure that they are removed from the sitempa submitted to Google; however after over a month we still see Google spider returning to the same pages and the webmaster tools shows no indicator that Google is shrinking our index size. Are there more effective and automated ways to make Google aware of a smaller index size in hope of Panda recovery? Potentially using the robots.txt file, GWT URL removal tool etc? Thanks /sp80