Webmaster Crawl errors caused by Joomla menu structure.

[dwallner]

Webmaster Tools is reporting crawl errors for pages that do not exist due to how my Joomla menu system works. Example, I have a menu item named "Service Area" that stores 3 sub items but no actual page for Service Area. This results in a URL like domainDOTcom/service-area/service-page.html

Because the Service Area menu item is constructed in a way that shows the bot it is a link, I am getting a 404 error saying it can't find domainDOTcom/service-area/ (The link is to "javasript:;") Note, the error doesn't say domainDOTcom/service-area/javascript:; it just says /service-area/

What is the best way to handle this? Can I do something in robots.txt to tell the bot that this /service-area/ should be ignored but any page after /service-area/ is good to go? Should I just mark them as fixed as it's really not a 404 a human will encounter or is it best to somehow explain this to the bot? I was advised on google forums to try this, but I'm nervous about it.

Disallow: /service-area/*
Allow: /service-area/summerlin-pool-service.
Allow: /service-area/north-las-vegas
Allow: /service-area/centennial-hills-pool-service

I tried a 301 redirect of /service-area to home page but then it pulls that out of the url and my landing pages become 404's.

http://www.lvpoolcleaners.com/

Thanks for any advice!

Derrick

[TomVolpe]

No problem Derrick, my pleasure.

Tom

[dwallner]

Wow,

Tom, thank you for the amazingly complete and well articulated response.   You, kind sir, are a interwebs Rock Star!

[TomVolpe]

Hi Derrick,

if you wish to use robots.txt you could simply use:

Allow: /service-area/*
Disallow: /service-area/

This will allow access to any child of /service-area/ but not /service-area/.

You could redirect this page to your homepage if you wished, and to stop children of this page being redirected you could use RedirectMatch instead of the Redirect directive and use a simple regular expression to only redirect if the URI ends with /service-area/, like this:

RedirectMatch 301 /service-area/?$ http://www.lvpoolcleaners.com/

The $ sign at the end signs that the apache should only redirect if the URI is ending in that pattern, and the ? after the trailing / allows the redirect to happen with or without the trailing slash.

But perhaps the simplest solution to this problem would be making your /service-area/ link point to '#' if the Joomla menu will allow it. This will append an empty anchor to the url, it will not refresh or redirect the page and anchors in URLs are not counted as duplicate URLs.

For human usability this would be the nicest way to interact with the menu, as you don't want a visitor being interrupted mid-way through their buying cycle by being sent back to the homepage when they didn't ask for it.

Hope that helps!