Huge google index with un-relevant pages

[stassaf]

Hi,

i run a site about sport matches, every match has a page and the pages are generated automatically from the DB. pages are not duplicated, but over time some look a little bit similar. after a match finishes it has no internal links or sitemap entry, but it's reachable by direct URL and continues to be on google index. so over time we have more than 100,000 indexed pages.

since past matches have no significance and they're not linked and a match can repeat and it may look like duplicate content....what you suggest us to do:

when a match is finished - not linked, but appears on the index and SERP

• 301 redirect the match Page to the match Category which is a higher hierarchy and is always relevant?

• use rel=canonical to the match Category

• do nothing....

*301 redirect will shrink my index status, some say a high index status is good...

*is it safe to 301 redirect 100,000 pages at once - wouldn't it look strange to google?

*would canonical remove the past matches pages from the index?

what do you think?

Thanks,

Assaf.

[Mark_Ginsberg]

In terms of what you've written, blocking a page via robots.txt doesn't remove it from the index. It simply prevents the crawlers from reaching the page. So if you block a page via robots.txt, the page remains in the index, Google just can't go back to the page and see if anything has changed. So if you were to block the page via robots.txt, and add a noindex tag to the page, Google won't be able to see the page with the noindex tag to remove it from the index because it's blocked via robots.txt.

If you moved all of your old content to a different folder, and block that folder via robots.txt, Google won't remove those pages from the index. In order to remove them from the index, you would have to go in to Webmaster Tools and use the URL removal tool to remove that new folder from the index - if they see it's blocked via robots.txt, then and only then they'll remove the content from the index - it has to be blocked via robots.txt first in order to remove the whole folder with the URL removal tool.

I'm not sure though if this would work for the future - if you removed a folder from the index, and then added more content that was indexed previously afterwards, I'm not sure what would happen to that new content moved to that folder. Either way, Google will have to come back and recrawl the page to see that it has moved to the new folder, and then remove it from the index. So either way, the content will only be removed once Google recrawls the old content.

So I still think a better way to remove the content from the index is to add the noindex tag to the old pages. To facilitate the search engines reaching these old pages, I'd make sure there is a way the engines can get to them - make sure there is a path they can take to reach them.

Another good idea I saw on a forum post here a while ago would be to create a sitemap containing all of these old pages you have indexed and want removed. Add the noindex tag to the sitemap - using the Webmaster tools sitemap interface, you'll then be able to monitor the progress of deindexation over time - by checking how many pages on the sitemap/s of the old content are originally indexed as reported by webmaster tools, and then you can see later on how many of those pages are still indexed, this will be a good indicator for you of the progress of the deindexation.

[stassaf]

Dear Mark,

*i've sent you a private message.

i'm starting to understand i've a much bigger problem.

*my index status contain 120k pages while only 2000 are currently relevant.

your suggestion is - after a match finishes pragmatically add  to the page and google will remove it from it's index. it could work for relatively new pages but since very old pages don't have links OR sitemap entry it could take a very long time to clear the index cause they're rarely crawled - if at all.

• more aggressive approach would be to change this site architecture and restrict by robot.txt the folder that holds all the past irrelevant pages.

so if today a match URL is like this: www.domain.com/sport/match/T1vT2

restrict www.domain.com/sport/match/ on robots.txt

and from now on create all new matches on different folder like: www.domain.com/sport/new-match-dir/T1vT2

• is this a good solution?

• wouldn't google penalize me for removing a directory with 100k pages?

• if it's a good approach, how much time it will take for google to clear all those pages from it's index?

I know it's a long one and i'll really appreciate your response.

Thanks a lot,

Assaf.

[Mark_Ginsberg]

there are a bunch of articles out there, but each case is different - here are a few:

http://www.seomoz.org/q/panda-recovery-what-is-the-best-way-to-shrink-your-index-and-make-google-aware

http://www.searchenginejournal.com/the-holy-grail-of-panda-recovery-a-1-year-case-study/45683/

You can contact me via private message here on the forum and I can try to take a more in depth look at your site if you can give me some more detailed info.

[stassaf]

yes. when the 1st Panda update was rolled out i've lost 50% of the traffic from google and haven't really recovered since.

[Mark_Ginsberg]

Are you sure you got hit by Panda before we talk about a Panda hit?

[stassaf]

Thanks Mark!

any good article about how to recover from Panda?

[Mark_Ginsberg]

Exactly - I'd build  a strategy more around promoting pages that will have long lasting value.

If you use the tag noindex, follow, it will continue to spread link juice throughout the site, it's just the individual page with the tag will not be included in the search results and will not be part of the index. In order for the tag to work, they first have to crawl the page and see the tag - so it doesn't happen instantaneously - if they crawl these deeper pages once every few weeks, once a month, or even longer, it may take a while for these pages to be removed from the index.

[stassaf]

Hi Mark

• these pages are very important when they are relevant (before the match finished) - they are the source of most of our traffic which come from long tail searches.

• some of these pages have inbound link and it would be a shame to lose all this juice.

• would noindex remove the pages from the google index? how much time it would take? wouldn't a huge noindex also look suspicious?

• by "evergreen pages" - you mean pages that are always relevant like League page / Sport page etc...?

Thanks,

Assaf.

[Mark_Ginsberg]

Hi Assaf,

(I'm not stalking you, I just think you've raised another interesting question)

In terms of index status/size, you don't want to create a massive index of empty/low value pages - this is food for Google's Panda algorithm, and will not be good for your site in the long run. It'll get a Panda smack if it hasn't already.

To remove these pages from the index, instead of doing hundreds of thousands of 301 redirects, which your server won't like either, I'd recommend adding the noindex meta tag to the pages.

I'd put a rule in your cms that after a certain point in time, you noindex those pages. Make sure you also have evergreen pages on your site that can serve as landing pages for the search engines and which won't need to be removed after a short period of time. These are the pages you'll want to focus your outreach and link building efforts on.

Mark