I don't think you can filter out on response codes.

However, first I would ensure you are running the right version of Java if you are on a 64bit machine.  The 32bit version functions but you cannot increase the memory allocation which is why you could be running into problems.  Take a look at http://www.screamingfrog.co.uk/seo-spider/user-guide/general/ under Memory.

You want Settings >> Show Advanced Settings >> (Privacy) Content Settings >> (Javascript) Do not allow any site to run javascript >> Finished.

Reload the site and check what you can see, or open up.

I've always stuck to Disallow and followed -

"This is currently a bit awkward, as there is no "Allow" field. The easy way is to put all files to be disallowed into a separate directory, say "stuff", and leave the one file in the level above this directory:"

http://www.robotstxt.org/robotstxt.html

From https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt this seems contradictory

| /* | equivalent to / | equivalent to / | Equivalent to "/" -- the trailing wildcard is ignored. |

I think this post will be very useful  for you - http://moz.com/community/q/allow-or-disallow-first-in-robots-txt

yep, sound good.

I was working on a site last year and they switched a DNN module based on your scenario without letting me know, having already tested the existing module.  First I saw was when rankings and traffic wobbled.  In this case the text was lost in the javascript and accounted for about 25-30% of content on all their main pages. Nightmare!

Took another look, also looked at documentation/online and don't see any way to exclude URLs from crawl based on response codes.  As I see it you would only want to exclude on name or directory as response code is likely to be random throughout a site and impede a thorough crawl.

I've just had fresh content crawled and indexed that is in this scenario.  Basically we are saying to the visitor "if you really want to know some more boring technical information then expand this, but we don't want to spoil your experience by vomiting all the data at you at once".  Crazy if that is changed.

You could legitimately have hidden content in terms of category/URLs - maybe only made available if you sign up as an example.  The guidelines are referring to manipulating on page content so there are elements within that page that are hidden and by default manipulating.

I have understood the length of the URL to be calculated post http:// unless it's https://.

Just the way it is displayed in the results.

I have run the site through ScreamingFrog.  You have masses and masses of 301 redirects going on which you need to resolve.  It looks like you could have some permalinks issue going on.  What do have set in Settings >> Permalinks?  I'd also look at installing Yoast Wordpress SEO as I recall there are options in there to handle permalinks as well.

Looks like you have been injected with spam links.  Used MajesticSEO to show the attached (sample).

I'd be assertive in getting into GWT asap.  I guess those URLs don't exist but it is still a big no.

http://i.imgur.com/XFNY1ZG.gif

If you go with the client disavow that domain for sure.

Without knowing the domain or any information in GWT nobody is going to be able to really help.

The only think I can think of is possibly .htaccess blocking Google.

IMHO directories in the main are dead.  Where they do succeed is when they are very niche and actually human managed.  If you have the skills and passion in your niche then there is a good possibility you will make it work and become a valuable resource.  Using NAP in the detail will provide valuable citation for the business listing.  SEO impact will be minimal, but for the business it is still a movement in the right direction.

You'll need to heavily market the directory to the business customer base as an invaluable one stop shop for their needs.  That will then drive referral traffic to the listings and allow you to charge a reasonable fee for inclusion.

I second ScreamingFrog.  Nothing else comes close.