Welcome to the Q&A Forum

Thanks for everything.

i'll stick to the slower method and see what's going on in the index.

Dear Dr. Meyers,

very insightful!!!

i must clear all the irrelevant page and the sooner the better.

(1) could take months or years

(2) sounds as a very good approach - i'm building my Sitemap with code so it's not a problem. the only problem is with a few hundreds at a time it could also take a long time. and wouldn't google spend a lot of time on crawling those pages and index less of the fresh new ones?

(3) what about google removal tool - and it's connected to my point on last post about setting a new site architecture:

• for all new matches=Pages create a new directory (without the irrelevant pages)
• ask WMT removal tool to remove the old directory and with it all the irrelevant pages (of course according to the guidelines for this tool)

what do you think about this approach?

Thanks again for all your help, i really appreciate it!

Assaf.

Dear Mark,

*i've sent you a private message.

i'm starting to understand i've a much bigger problem.

*my index status contain 120k pages while only 2000 are currently relevant.

your suggestion is - after a match finishes pragmatically add  to the page and google will remove it from it's index. it could work for relatively new pages but since very old pages don't have links OR sitemap entry it could take a very long time to clear the index cause they're rarely crawled - if at all.

• more aggressive approach would be to change this site architecture and restrict by robot.txt the folder that holds all the past irrelevant pages.

so if today a match URL is like this: www.domain.com/sport/match/T1vT2

restrict www.domain.com/sport/match/ on robots.txt

and from now on create all new matches on different folder like: www.domain.com/sport/new-match-dir/T1vT2

• is this a good solution?

• wouldn't google penalize me for removing a directory with 100k pages?

• if it's a good approach, how much time it will take for google to clear all those pages from it's index?

I know it's a long one and i'll really appreciate your response.

Thanks a lot,

Assaf.

Dear Dr. Meyers,

i'm starting to understand i've a much bigger problem.

all finished matches are not relevant anymore and though you can reach them (their Page) from SERP or direct URL, they don't appear on site links OR sitemap. so the best idea is to remove all these old pages from google index - they don't contribute + they made my index status contain 120k pages while only 2000 are currently relevant.

this causes waste of google crawling on irrelevant pages and a potential that google may see some of them as dupes cause in some cases most of the page is relatively similar.

one suggestion i got is - after a match finishes pragmatically add  to the page and google will remove it from it's index. - will it remove it if there're no links/sitemap to this page???

but i also have to handle the problem of the huge index - the above approach may/or not handle pages from now on, but what about all the other far past pages with finished matches??? how can i remove them all from the index.

• adding  <meta name="robots" content="noindex,follow">to all of them could take months or more to clean the index cause they're probably rarely crawled.</meta name="robots" content="noindex,follow">

• more aggressive approach would be to change this site architecture and restrict by robot.txt the folder that holds all the past irrelevant pages.

so if today a match URL is like this: www.domain.com/sport/match/T1vT2

restrict www.domain.com/sport/match/ on robots.txt

and from now on create all new matches on different folder like: www.domain.com/sport/new-match/T1vT2

• is this a good solution?

• wouldn't google penalize me for removing a directory with 100k pages?

• if it's a good approach, how much time it will take for google to clear all those pages from it's index?

I know it's a long one and i'll really appreciate your response.

Thanks a lot,

Assaf.

yes. when the 1st Panda update was rolled out i've lost 50% of the traffic from google and haven't really recovered since.

Thanks Mark!

any good article about how to recover from Panda?

Hi Mark

• these pages are very important when they are relevant (before the match finished) - they are the source of most of our traffic which come from long tail searches.

• some of these pages have inbound link and it would be a shame to lose all this juice.

• would noindex remove the pages from the google index? how much time it would take? wouldn't a huge noindex also look suspicious?

• by "evergreen pages" - you mean pages that are always relevant like League page / Sport page etc...?

Thanks,

Assaf.

Hi,

i run a site about sport matches, every match has a page and the pages are generated automatically from the DB. pages are not duplicated, but over time some look a little bit similar. after a match finishes it has no internal links or sitemap entry, but it's reachable by direct URL and continues to be on google index. so over time we have more than 100,000 indexed pages.

since past matches have no significance and they're not linked and a match can repeat and it may look like duplicate content....what you suggest us to do:

when a match is finished - not linked, but appears on the index and SERP

• 301 redirect the match Page to the match Category which is a higher hierarchy and is always relevant?

• use rel=canonical to the match Category

• do nothing....

*301 redirect will shrink my index status, some say a high index status is good...

*is it safe to 301 redirect 100,000 pages at once - wouldn't it look strange to google?

*would canonical remove the past matches pages from the index?

what do you think?

Thanks,

Assaf.

we're on the same stage.

tips and ideas please

Thanks!

CuteRank - a small desktop app that can be updated on demand - shows top 100

http://cuterank.net/

1st thing is to add google authorship (if you haven't already done it)

https://plus.google.com/authorship

i've got some good responses, but i'm not sure what to do.

any other opinions will be highly appreciated.

Thanks!

Hi Dr. Meyers,

thanks for your detailed response.

just wanted to refine my scenario:

1. the case of pairs (repeat match after a short term) is rare, but i encountered it.

2. there're no links or sitemap entry for the match that already finished. but google keeps it in the index. the page is reachable ONLY by direct URL address or from the SERP.

3. i don't think i can enforce google to automatically remove the old match from the index and doing it manually for 1000's of matches is not an option.

4. i thought google recognize the content of each page to determine if it's duplicate and not only by the URL/title - by tool the content is only 66% similar.

5. currently i've this problem twice - so for one case i've made rel=canonical and the other one i'm letting google to decide. when google encounters a rel=canonical does it goes to the URL of the canonical?

Thanks,

Assaf.

Thanks Jesse!

1. the content is different - according to a comparison tool they are 64% similar and considering the menus, header of the site and other element that appears on each page - you can say they're unique - don't i? even so google haven't indexed the 2nd page and it's up for 5 days - sitemap indexing rate is 90% according to google webmaster tools. so what wrong here?

2. including the date seems like a good idea! but 2 questions about it:

-won't the URL look messy with these numeric inputs?

• the same same match can be repeated in the future isn't it a good idea that the page is already indexed? i mean the URL will stay the same, just the content will be different.

Thanks,

Assaf.

Highland thanks for your quick response.

the pages are created dynamically cause at every moment we have more then 1000 matches on our DB. it's impossible to create a manual URL for each page.

the case i described is rare, but it happened for a very important match.

Hi,

i've a site about sport which cover matches. for each match i've a page.

last week there was a match between: T1 v T2

so a page was created:

www.domain.com/match/T1vT2   - Page1

this week T2 host T1, so there's a new page

www.domain.com/match/T2vT1   - Page2

each page has a unique content with Authorship, but the URL, Title, Description, H1

look very similar cause the only difference is T2 word before T1.

though Page2 is available for a few days, on site links & sitemap, for the search query "T2 T1 match" Page1 appears on the SERP (high location).

of course i want Page2 to be on SERP for the above query cause it's the relevant match.

i even don't see Page2 anywhere on the SERP and i think it wasn't indexed.

Questions:

1. do you think google see both pages as duplicated though the content is different?

2. is there a difference when you search for

T1 vs T2

OR

T2 vs T1

?

3. should i redirect 301 Page1 to Page2? consider that all content for Page1 and the Authorship G+ will be lost.

4. should i make rel=canonical on Page1 to Page2?

5. should i let google sort it out?

i know it's a long one, thanks for your patience.

Thanks,

Assaf