Why do old URL format are still being crawled by Rogerbot?

[Trigun]

Hi,

In the early days of my blog, I used permalinks with the following format:

http://www.mysitesamp.com/2009/02/04/heidi-cortez-photo-shoot/

I then decided to change this format using .htaccess to this format:

http://www.mysitesamp.com//heidi-cortez-photo-shoot/

My question is, why do rogerbot still crawls my old URL format since these urls' no longer exists in my website or blog.

[Trigun]

Thanks Alan,

That solved my problem...

[AlanMosley]

Here is an example
www.lugaluda.com/new-2ne1-i-dont-care-music-video-i-dont-care-lyrics/
has a link to www.lugaluda.com/2009/07/09/new-2ne1-i-dont-care-music-video-i-dont-care-lyrics/

[Trigun]

Hi Alan,

After disallowing the directory in robots.txt, Rogerbot still includes the non-existing URLs.  Here is a sample URL that is being reported by Rogerbot

www.lugaluda.com/2009/08/05/chase-online-banking-chase-checking-bonus/

[AlanMosley]

If you give me the url, i can crawl it fior you if you like.

[Trigun]

Thanks Alan, I really appreciate your help.  Gave me an idea since all the old URLs are coming from a virtual 2009 directory,  I tried to add a disallow statement for that directory in the robots.txt section.  Hopefully this will help solve the problem.

I will let you know the results after rogerbot finishes recrawling my site...

Thanks Dude....

[AlanMosley]

You need to search your site, but bots start on a page and follow the links, if the report them then they must of found them, bots like googlebot or bingbot can find them on other sites, but rogerbot is only crawling within your site.

[Trigun]

How will I know if they still exists on my site?  If I tried to access the specific URLs, they are no longer active.

[AlanMosley]

The old format must still exist in your site somewhere, bots follow links from your home page though your site.