External Links from own domain

[JacoRoux]

Hi all,

I have a very weird question about external links to our site from our own domain.

According to GWMT we have 603,404,378 links from our own domain to our domain (see screen 1) We noticed when we drilled down that this is from disabled sub-domains like m.jump.co.za.

In the past we used to redirect all traffic from sub-domains to our primary www domain. But it seems that for some time in the past that google had access to crawl some of our sub-domains, but in december 2010 we fixed this so that all sub-domain traffic redirects (301) to our primary domain. Example http://m.jump.co.za/search/ipod/ redirected to http://www.jump.co.za/search/ipod/

The weird part is that the number of external links kept on growing and is now sitting on a massive number.

On 8 April 2011 we took a different approach and we created a landing page for m.jump.co.za and all other requests generated 404 errors. We added all the directories to the robots.txt and we also manually removed all the directories from GWMT.

Now 3 weeks later, and the number of external links just keeps on growing: Here is some stats:

11-Apr-11 - 543 747 534

12-Apr-11 - 554 066 716

13-Apr-11 - 554 066 716

14-Apr-11 - 554 066 716

15-Apr-11 - 521 528 014

16-Apr-11 - 515 098 895

17-Apr-11 - 515 098 895

18-Apr-11 - 515 098 895

19-Apr-11 - 520 404 181

20-Apr-11 - 520 404 181

21-Apr-11 - 520 404 181

26-Apr-11 - 520 404 181

27-Apr-11 - 520 404 181

28-Apr-11 - 603 404 378

I am now thinking of cleaning the robots.txt and re-including all the excluded directories from GWMT and to see if google will be able to get rid of all these links.

What do you think is the best solution to get rid of all these invalid pages.

moz1.PNG moz2.PNG moz3.PNG

[JacoRoux]

We had 301s for about 6 months, and the old URLs did not disappear from google. Thats why we decided to change them to 404s, with the thinking that Google might remove them quicker. But the number of links from sub-domains just keeps on growing.

I am worried that by having these problem urls listed in the robots.txt actually prevents google from following them and seeing that it should be removed and that it returns a 404

[Getz.pro]

Instead of trying to manage a massive 301 list, can you just customize your 404 page to redirect?

{script to test page URL}

$location = "http://www.YourSite.com/";

header("HTTP/1.1 301 Moved Permanently");

header("Location: {$location}");

exit;

}

[JacoRoux]

Update:

There are 2 things that still puzzles me with this:

If you go to http://www.google.co.za/search?q=site:jump.co.za+-www&hl=en&rlz=1C1GPCK_enZA426ZA426&prmd=ivns&filter=0&biw=1920&bih=979 you notice all sorts of weird sub-domains, and all of these are invalid and have been removed from GWMT.

If you manage the domain m.jump.co.za on GWMT you also notice that it still reports on keywords, queries and all sorts of data, although the site is disabled and all the URLs generate 404 errors

There is only a few of these weird sub-domains that are causing the problems:

0www.
iiiiiwww.
iwww.
m.
wtfwww.
www.www.
wwww.

All these domains feels very fimiliar to me and I am almost 100% sure that its domains that used to test when we found the problem on apache, meaning google took the data from the toolbar queries and probably started indexing these sub-domains. But now I can't get rid of them, and Google seems to be out of control with these.

So the main question is probably, should we just give 404s or should we add to Robots.txt as well?