Welcome to the Q&A Forum

The sites are massive and we are talking massive numbers:

Google reports in WMT that site B still has 259,157,970 links to site A, although when you filter into the report it only shows a few

The current state is that nothing is blocked on Site B, and ALL pages point to the landing page of Site B.

In WMT for site B, G still shows data for all the reports, like search queries, keywords, crawl errors (very old and all fixed)  and so on. The reports and data does not bother me as much as the 259,157,970 links it reports on Site A.

On the 11th of April when I started the process of getting rid of these links, there were 554,066,716, this jumped up to 603,404,378 on the 28th of April. It started dropping and was as low as 122,405,100 on the 17th of May, and then started growing again up to where it is now 259,157,970

I also noticed that when the pages was giving 404s that the crawl rate of google dropped to zero, now that its redirecting to the landing page, the crawl rate is back up to about 1,800 per day, which is still very low, considering the numbers we are talking about.

The crawl rate on Site A is okay, at 220,000 per day, but it was as high as 800,000 per day at one stage.

I failed to mention that both sites A and B had the exact same content, database and URL structure, with the only difference being the sub domain.

Hi,

I have two sites, lets call them site A and site B, both are sub domains of the same root domain. Because of a server config error, both got indexed by Google.

Google reports millions of inbound links from Site B to Site A

I want to get rid of Site B, because its duplicate content.

First I tried to remove the site from webmaster tools, and blocking all content in the robots.txt for site B, this removed all content from the search results, but the links from site B to site A still stayed in place, and increased (even after 2 months)

I also tried to change all the pages on Site B to 404 pages, but this did not work either

I then removed the blocks, cleaned up the robots.txt and changed the server config on Site B so that everything redirects (301) to a landing page for Site B. But still the links in Webmaster Tools to site A from Site B is on the increase.

What do you think is the best way to delete a site from google and to delete all the links it had to other sites so that there is NO history of this site? It seems that when you block it with robots.txt, the links and juice does not disappear, but only the blocked by robots.txt report on WMT increases

Any suggestions?

We had 301s for about 6 months, and the old URLs did not disappear from google. Thats why we decided to change them to 404s, with the thinking that Google might remove them quicker. But the number of links from sub-domains just keeps on growing.

I am worried that by having these problem urls listed in the robots.txt actually prevents google from following them and seeing that it should be removed and that it returns a 404

Update:

There are 2 things that still puzzles me with this:

If you go to http://www.google.co.za/search?q=site:jump.co.za+-www&hl=en&rlz=1C1GPCK_enZA426ZA426&prmd=ivns&filter=0&biw=1920&bih=979 you notice all sorts of weird sub-domains, and all of these are invalid and have been removed from GWMT.

If you manage the domain m.jump.co.za on GWMT you also notice that it still reports on keywords, queries and all sorts of data, although the site is disabled and all the URLs generate 404 errors

There is only a few of these weird sub-domains that are causing the problems:

0www.
iiiiiwww.
iwww.
m.
wtfwww.
www.www.
wwww.

All these domains feels very fimiliar to me and I am almost 100% sure that its domains that used to test when we found the problem on apache, meaning google took the data from the toolbar queries and probably started indexing these sub-domains. But now I can't get rid of them, and Google seems to be out of control with these.

So the main question is probably, should we just give 404s or should we add to Robots.txt as well?

Hi all,

I have a very weird question about external links to our site from our own domain.

According to GWMT we have 603,404,378 links from our own domain to our domain (see screen 1) We noticed when we drilled down that this is from disabled sub-domains like m.jump.co.za.

In the past we used to redirect all traffic from sub-domains to our primary www domain. But it seems that for some time in the past that google had access to crawl some of our sub-domains, but in december 2010 we fixed this so that all sub-domain traffic redirects (301) to our primary domain. Example http://m.jump.co.za/search/ipod/ redirected to http://www.jump.co.za/search/ipod/

The weird part is that the number of external links kept on growing and is now sitting on a massive number.

On 8 April 2011 we took a different approach and we created a landing page for m.jump.co.za and all other requests generated 404 errors. We added all the directories to the robots.txt and we also manually removed all the directories from GWMT.

Now 3 weeks later, and the number of external links just keeps on growing: Here is some stats:

11-Apr-11 - 543 747 534

12-Apr-11 - 554 066 716

13-Apr-11 - 554 066 716

14-Apr-11 - 554 066 716

15-Apr-11 - 521 528 014

16-Apr-11 - 515 098 895

17-Apr-11 - 515 098 895

18-Apr-11 - 515 098 895

19-Apr-11 - 520 404 181

20-Apr-11 - 520 404 181

21-Apr-11 - 520 404 181

26-Apr-11 - 520 404 181

27-Apr-11 - 520 404 181

28-Apr-11 - 603 404 378

I am now thinking of cleaning the robots.txt and re-including all the excluded directories from GWMT and to see if google will be able to get rid of all these links.

What do you think is the best solution to get rid of all these invalid pages.

moz1.PNG moz2.PNG moz3.PNG