Webmaster Crawl errors caused by Joomla menu structure.
-
Webmaster Tools is reporting crawl errors for pages that do not exist due to how my Joomla menu system works. Example, I have a menu item named "Service Area" that stores 3 sub items but no actual page for Service Area. This results in a URL like domainDOTcom/service-area/service-page.html
Because the Service Area menu item is constructed in a way that shows the bot it is a link, I am getting a 404 error saying it can't find domainDOTcom/service-area/ (The link is to "javasript:;") Note, the error doesn't say domainDOTcom/service-area/javascript:; it just says /service-area/
What is the best way to handle this? Can I do something in robots.txt to tell the bot that this /service-area/ should be ignored but any page after /service-area/ is good to go? Should I just mark them as fixed as it's really not a 404 a human will encounter or is it best to somehow explain this to the bot? I was advised on google forums to try this, but I'm nervous about it.
Disallow: /service-area/*
Allow: /service-area/summerlin-pool-service.
Allow: /service-area/north-las-vegas
Allow: /service-area/centennial-hills-pool-serviceI tried a 301 redirect of /service-area to home page but then it pulls that out of the url and my landing pages become 404's.
http://www.lvpoolcleaners.com/
Thanks for any advice!
Derrick
-
No problem Derrick, my pleasure.
Tom
-
Wow,
Tom, thank you for the amazingly complete and well articulated response. You, kind sir, are a interwebs Rock Star!
-
Hi Derrick,
if you wish to use robots.txt you could simply use:
Allow: /service-area/*
Disallow: /service-area/This will allow access to any child of /service-area/ but not /service-area/.
You could redirect this page to your homepage if you wished, and to stop children of this page being redirected you could use RedirectMatch instead of the Redirect directive and use a simple regular expression to only redirect if the URI ends with /service-area/, like this:
RedirectMatch 301 /service-area/?$ http://www.lvpoolcleaners.com/
The $ sign at the end signs that the apache should only redirect if the URI is ending in that pattern, and the ? after the trailing / allows the redirect to happen with or without the trailing slash.
But perhaps the simplest solution to this problem would be making your /service-area/ link point to '#' if the Joomla menu will allow it. This will append an empty anchor to the url, it will not refresh or redirect the page and anchors in URLs are not counted as duplicate URLs.
For human usability this would be the nicest way to interact with the menu, as you don't want a visitor being interrupted mid-way through their buying cycle by being sent back to the homepage when they didn't ask for it.
Hope that helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Issues with structured data on angular pages.
I am having an issue with Structured Data. I have added the structured data to angular pages of my site but when I run the test from the testing tool it doesn't detect the same. Although when I cut and paste the code (from inspect element) it detects the structured data. But in my webmaster tools, those pages don't show up under structured data. I am unsure if my structured data is being picked up by google. What should be done here? Should I provide pre-rendered pages to google?
Technical SEO | | Lybrate06060 -
Increase in pages crawled per day
What does it mean when GWT abruptly jump from 15k to 30k pages crawled per day? I am used to see spikes, like 10k average and a couple of time per month 50k pages crawled. But in this case 10 days ago moved from 15k to 30k per day and it's staying there. I know it's a good sign, the crawler is crawling more pages per day, so it's picking up changes more often, but I have no idea of why is doing it, what good signals usually drive google crawler to choose to increase the number of pages crawled per day? Anyone knows?
Technical SEO | | max.favilli1 -
Responsive web design has a crawl error of redirecting to HTTP instead of HTTPS ? is this because of the new update of google that appreciates the HTTPs more?
We at yamsafer.me are using a Repsonsive web design! A crawl errors occured which redirects the hompage to an HTTP version instead of HTTPS? Any ideas on why this happened?
Technical SEO | | Yamsafer.com0 -
Pages appear fine in browser but 404 error when crawled?
I am working on an eCommerce website that has been written in WordPress with the shop pages in E commerce Plus PHP v6.2.7. All the shop product pages appear to work fine in a browser but 404 errors are returned when the pages are crawled. WMT also returns a 404 error when ‘fetch as Google’ is used. Here is a typical page: http://www.flyingjacket.com/proddetail.php?prod=Hepburn-Jacket Why is this page returning a 404 error when crawled? Please help?
Technical SEO | | Web-Incite0 -
429 Errors?
I have over 500,000 429 errors in webmaster tools. Do I need to be concerned about these errors?
Technical SEO | | TheKrazyCouponLady0 -
Regarding Structured Data
I was wondering if you have any insight as to why my star ratings are yet to show up in google SERPs. I've set up a test (months ago) just to test the formatting, my code looks like this: <span typeof="<a class="attribute-value">v:Review-aggregate</a>" xmlns:v="<a class="attribute-value">http://rdf.data-vocabulary.org/#</a>">There are currently <span property="<a class="attribute-value">v:count</a>">15span> reviews for <span property="<a class="attribute-value">v:itemreviewed</a>">Ford Mustang brakesspan> with an average rating of <span rel="<a class="attribute-value">v:rating</a>"><span typeof="<a class="attribute-value">v:Rating</a>"><span property="<a class="attribute-value">v:average</a>">4.50span> out of <span property="<a class="attribute-value">v:best</a>">5span> stars.span>span>span> Any ideas why it hasn't show up yet?
Technical SEO | | andrewv0 -
Domain structure for US Local Sites
We are planning on opening localized versions of our website throughout the world and in the US. For countries these websites will be: www.site.co.uk www.site.fr etc.... For the US would it be better to add the states onto part of the domain name or use a sub-folder. What is the advantage/disadvantages of each? Meaning, should it be: nj.site.com or site-nj.com
Technical SEO | | theLotter0 -
Help with strange 404 Errors.
For the most part I have never had trouble tracking down 404's. Usually it's simply a broken link, but lately I have been getting these strange errors http://gridironexperts.com/http%3A/www.nfl.com/gamecenter?game_id=29528&season=2008&displayPage=tab_gamecenter/ What does; %C2%94 repersent? The error always points to NFL.com, but we don't link to them...like ever? Can I just 404: http://gridironexperts.com// to fix the problem, as all 404's start with this weird %C2%94 error. Is this error even on my site? Is in the backend...virus? thanks -Mike
Technical SEO | | MikePatch0