Dynamic URL pages in Crawl Diagnostics

[Visually]

The crawl diagnostic has found errors for pages that do not exist within the site. These pages do not appear in the SERPs and are seemingly dynamic URL pages.

Most of the URLs that appear are formatted http://mysite.com/keyword,%20_keyword_,%20key_word_/ which appear as dynamic URLs for potential search phrases within the site.

The other popular variety among these pages have a URL format of http://mysite.com/tag/keyword/filename.xml?sort=filter which are only generated by a filter utility on the site.

These pages comprise about 90% of 401 errors, duplicate page content/title, overly-dynamic URL, missing meta decription tag, etc. Many of the same pages appear for multiple errors/warnings/notices categories.

So, why are these pages being received into the crawl test? and how to I stop it to gauge for a better analysis of my site via SEOmoz?

[cmaseattle]

I am having a similar issue. I am getting hit with 404 errors for pages that do not exist anymore of have been fixed. How do I get these to stop showing up?

[cmaseattle]

I am having a similar issue. I am getting hit with 403 errors for pages that do not exist anymore of have been fixed. How do I get these to stop showing up?

[RobertFisher]

Based on what has happened from time to time on our sites, my guess will be that it is caused by a widget or plug in on your CMS in some way interacting with the Bot. You are likely being crawled on these urls by Google (and producing 404's) as well and it is not likely it is just Roger bot picking it up. There is a lot on the GWMT forums regarding this with a myriad of suggested fixes: mod rewrite, http 410 for 404, etc.

One fix used by many is if your site has relative links you can do full out urls. If you have a ton of pages this might be a bit more of a pain. (Our clients typically have smaller sites so not too much of a problem).

If you are using WordPress (or another CMS that can utilize Extra Options Plug In) it is stated in the forums that the 404's can be stopped by:

In Extra Options plugin: I checked off all of the below options,, the last two do the job..    read about the nonindex nonfollow where appropriate,,, in that plugin,, this could be the answer.

Make meta descriptions from excerpts         
Make home meta description from tagline

Add noindex where appropriate  
Add nofollow where appropriate

Another option is to insure you have no

There are plenty of bright coders on the moz who can pitch in here and be more eloquent,

Hope this helps.