Welcome to the Q&A Forum

I'm trying to add a URL that is having some obvious issues so I can further investigate. When trying to add this site to a campaign in SEOmoz i get the following:

Roger has detected a problem:

We have detected that the root domain theurbandater.com is not a live URL. Using this domain, we will be unable to crawl your site or present accurate SERP information.

==

What does that error mean? Where should I be looking to begin troubleshooting?

The initial issue was that back on 9/1 according to Google Webmaster Tools this site began getting a high number of 500 erros and that number continued to rise up to 3200 of the same type of error. So something screwy is going on and I'm not sure where to start looking.

Wow guys! Thanks for the responses. You've given me a couple things to think about.

The suggestions from Thomas and GYMSN might be the way I choose to plod ahead. Though, the rewrite rule seems like an interesting play. I have no idea what either of those lines mean. Heh. Is there a reference you could recommend, Woj, that might shed more light on rewrite rules?

A couple technical pieces to throw out there:

1. It's a WordPress site, so they don't have "folders," per se.

2. They run on linux, with an apache web server

I have a client who's experiencing a number of crawl errors, which I've gotten down fo 9,000 from 18,000. One of the challenges they experience is that they've modified their URL structure a couple times.

First it was: site.com/year/month/day/post-name
Then it was: site.com/category/post-name
Now it's: site.com/post-name

I'm not sure of the time elapsed between these changes, but enough time has passed that the URLs for the previous two URL structures have been indexed and spit out 404s now.

What's the best/clean way to address this issue?I'm not going to create 9k redirect rules obviously, but there's got to be a way to address this issue and resolve it moving forward.

Awesome! Thanks for clarifying that.

I'm assuming I'd us a rule similar to below. Is that right/

RewriteEngine on
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_FILENAME}\.html -f
RewriteRule ^(.*)$

THey are just seeing sites without them and also would like to not have them.

That's about it, really. Nothing is broken or having issues being indexed.

Client wishes to drop the .php extension on all their pages (they've got around 2k pages). I assured them that wasn't necessary. However, in the event that I do end up doing this what's the best practices way (and easiest way) to do this? This is also a WordPress site.

Thanks.

I want to bring my categories down to about 8 or so and the tags... They're just a mess and I'd really like to bring that figure down significantly and setup a standard for usage.

My thought was to remove the un-needed tags and categories and setup 301 redirects for the ones that I'm removing.

Is that even necessary?

Are there tools that can assist with this?

What are the "gotchas" I should be aware of?

Thanks!

I do understand that I can at a robots.txt rule. I just want to make sure my understanding was correct. If a page is noindexed, it should be ignored by search engines, right?

I've got several pages that have "too many on page links" and the pages mentioned have already been noindexed. Do these pages need to be no followed too? Here's one of the pages: http://digisavvy.com/site-map/.

There's several pages like this, most of which are category or tag archives, which I've noindexed... Do I need to nofollow these, too?

I want to bring my categories down to about 8 or so and the tags... They're just a mess and I'd really like to bring that figure down significantly and setup a standard for usage.

My thought was to remove the un-needed tags and categories and setup 301 redirects for the ones that I'm removing.

Is that even necessary?

Are there tools that can assist with this?

What are the "gotchas" I should be aware of?

Thanks!