How to handle Not found Crawl errors?
-
I'm using Google webmaster tools and able to see Not found Crawl errors. I have set up custom 404 page for all broken links. You can see my custom 404 page as follow.
http://www.vistastores.com/404
But, I have question about it.
Will it require to set 301 redirect for broken links which found in Google webmaster tools?
-
I agree with Ben on this one. There are plenty of 404s caused by scraper sites that don't and won't affect my time, especially on big sites.
Also, redirects aren't the only tool available. There are plenty of other ways to fix GWT 404 errors, particularly if there is a fundmental problem aside from the link in question.
-
Hi Commerce, I was certainly came across a blog post on this topic on Google's Webmaster Central blog, it covers most of the questions around 404 errors.
Generally speaking:
- If these are pages that you removed, then the 404 HTTP result code is fine.
- If these are pages that changed addresses, then you should 301 redirect to the new addresses. How you do this depends on your setup, for Apache-servers you may be able to use the .htaccess file for this.
- Unless these are pages that used to receive a lot of traffic from search, these 404s won't be the reason for your site's traffic dropping like that. Google understands that the web changes and that URLs disappear - that is not a reason for Google to stop showing your site.
So my recommendation would be to check the URLs that are listed as 404 crawl errors. If any are important, then set up redirects to the appropriate new URLs as soon as you can. If none of them are important, then keep this in mind as something worth cleaning up when you have time, but focus on the rest of your site first. Often drastic drops in traffic are due more to the general quality of the website, so that's what I'd recommend working on first.
For more deatails refer to How to Fix Crawl Errors.
I hope that your query had been solved.
-
Makes sense - in which case the homepage might not be the best place for you.
Another option for the custom 404 which works well in certain circumstances is to add a dynamic element to it.
For example, we know the referring URL has reference to product XYZ which may now be unavailable, but perhaps we can dynamically pull in other relevant products into the 404 page.
Thats something I am looking to do with hotels that become unavailable - pull in a dynamic element to the 404 which basically recommends some other hotels close by.
-
Well I would have to disagree with that principal. Sometimes you have to think a little broader than just SEO and ask yourself if it really makes commercial sense to redirect everything.
That's why I put a financial cost against each unique redirect. At the end of the day it requires someone to action it and that person has a cost associated with their time that may be better allocated working on something that will actually drive business uplift or improve customer experience.
Each to their own of course, but I see a lot of SEO's who don't think big picture and they up using up developer resource doing stuff that then has no impact. It just p!sses people off in my experience.
-
Hi Ben,
I agree with you that some links are not worth redirecting. However, in my experience a dead link never comes alone. Often there is some kind of reason that the link was created, and there might be others you don't know about.
For this reason I usually recommend redirecting all broken links, even if the individual link is not worth the trouble. Obviously there are exceptions to this rule, but most of the time it's worth your trouble.
Sven
-
Good to know! But, I have very bad experience to redirect such a strong page to home page. I have removed too many product pages for market umbrellas from my website and redirect it to home page. Because, I don't have specific landing page or inner level page for it. So, I'm able to see change over ranking for specific keywords. My home page is ranking well in Market Umbrellas keyword because too many external page link my product page with that keyword. It also create negative ranking impression for my actual targeted keyword which I'm using for my home page.
-
Yeah, which is basically what Kane is saying as well. If you don't have an appropriate internal page then you could send the 301 redirect to your homepage or if it was a specific product you might want to redirect it to the parent/child category.
If its a particularly strong URL that has been linked to from many good external sources then you might consider adding a replacement content page and redirecting to that.
Ben
-
Hi Ben,
I got your point. If my page is available on external page which have good value (Good page rank or heavy amount of traffic) so, I need to redirect it on specific internal page to save my page rank flow. Right?
-
Hopefully I am understanding your question correctly here....
The main benefit of the custom 404 page aside from the obvious improvement to user experience is that you provide additional links into content that otherwise wouldn't necessarily be available to the search bots.
In essence if you just had a standard 404 error page you'd send the search bots to a dead page where their only decision would be to leave the domain and go elsewhere.
Regards setting up 301 redirects I like to associate a cost to each 301 redirect. Imagine the time it will take you or someone else to set each redirect up (say $5 per redirect). Then consider the following:
Is the URL that is 404 worth redirecting?
(1) Does it hold some residual SEO value (i.e., is it present on external sites that is driving link equity? if so can you redirect that equity to somewhere more valuable?
(2) Is the URL present on an external site driving referral traffic? if so do you have a new content page that will still match the users intent?
if the URL(s) that are 404'ing have no real link equity associated to them and/or you don't have a genuinely useful page to redirect the user to then I would just let them hit the 404 page.
If in doubt put yourself in a users boots and ask yourself if the set-up you have done would offer a valuable experience? no point redirecting a user to something totally irrelevant to the original intent - it'll just p!ss them off most the time and increase your bounce rate.
-
If there is a link pointed at that 404 page, then I will almost always 301 it to regain that link value. If I control the source of the link, I'll change that instead. If the link is from a spammy or junky website, I don't worry about it.
Here is a worthwhile article on how to go about fixing GWT crawl errors:
http://www.seomoz.org/blog/how-to-fix-crawl-errors-in-google-webmaster-tools
I would suggest adding more content to your 404 page. Try to help people find what they're looking for by suggesting common pages, product segments, etc.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How to handle SEO redirection after remove www
Recently our https://bestclassifiedsusa.com free classifieds USA website we move from www to non www format. Now old urls still showing in Google SERP. When click they show 404 error pages due to removed old ads.
Technical SEO | | bcuclassifieds
We redirect www to non www version for main domain but still old cache pages affected.
How to dynamically handle this case, all old www version ads pages auto redirect to relevant category pages?0 -
What should I do with URLs that cause site map errors?
Hi Mozzers, I have a client who uses an important customer database and offers gift cards via https://clients.mindbodyonline.com located within the navigation which causes sitemap errors whenever it is submitted since domain is different. Should I ask to remove those links from navigation? if so where can I relocate those links? If not what should I do to have a site map without any errors? Thanks! 1n16jlL.png
Technical SEO | | Ideas-Money-Art0 -
How can I best handle parameters?
Thank you for your help in advance! I've read a ton of posts on this forum on this subject and while they've been super helpful I still don't feel entirely confident in what the right approach I should take it. Forgive my very obvious noob questions - I'm still learning! The problem: I am launching a site (coursereport.com) which will feature a directory of schools. The directory can be filtered by a handful of fields listed below. The URL for the schools directory will be coursereport.com/schools. The directory can be filtered by a number of fields listed here: Focus (ex: “Data Science”) Cost (ex: “$<5000”) City (ex: “Chicago”) State/Province (ex: “Illinois”) Country (ex: “Canada”) When a filter is applied to the directories page the CMS produces a new page with URLs like these: coursereport.com/schools?focus=datascience&cost=$<5000&city=chicago coursereport.com/schools?cost=$>5000&city=buffalo&state=newyork My questions: 1) Is the above parameter-based approach appropriate? I’ve seen other directory sites that take a different approach (below) that would transform my examples into more “normal” urls. coursereport.com/schools?focus=datascience&cost=$<5000&city=chicago VERSUS coursereport.com/schools/focus/datascience/cost/$<5000/city/chicago (no params at all) 2) Assuming I use either approach above isn't it likely that I will have duplicative content issues? Each filter does change on page content but there could be instance where 2 different URLs with different filters applied could produce identical content (ex: focus=datascience&city=chicago OR focus=datascience&state=illinois). Do I need to specify a canonical URL to solve for that case? I understand at a high level how rel=canonical works, but I am having a hard time wrapping my head around what versions of the filtered results ought to be specified as the preferred versions. For example, would I just take all of the /schools?focus=X combinations and call that the canonical version within any filtered page that contained other additional parameters like cost or city? Should I be changing page titles for the unique filtered URLs? I read through a few google resources to try to better understand the how to best configure url params via webmaster tools. Is my best bet just to follow the advice on the article below and define the rules for each parameter there and not worry about using rel=canonical ? https://support.google.com/webmasters/answer/1235687 An assortment of the other stuff I’ve read for reference: http://www.wordtracker.com/academy/seo-clean-urls http://www.practicalecommerce.com/articles/3857-SEO-When-Product-Facets-and-Filters-Fail http://www.searchenginejournal.com/five-steps-to-seo-friendly-site-url-structure/59813/ http://googlewebmastercentral.blogspot.com/2011/07/improved-handling-of-urls-with.html
Technical SEO | | alovallo0 -
Should we handle this redirect differently?
So our question is should we handle page redirection/rewriting in php or in .htaccess (with a specific problem we are running into outlined below). We have an ecommerce store in a subfolder of our site (example.com/store/). In the next folder down we have a group of widgets(www.example.com/store/widget-group1). Recently we put a .htaccess redirect in the top level folder (example.com/store/.htaccess), in order to re-write some URL’s and also 301 a page to another page. This seems to be negatively affecting our /widgets-group1/ subfolder however (organic traffic to example.com/store/widget-group1) took a nose dive 3 days after putting the .htaccess redirect in place on the /store/ folder and it has not recovered 8 days later). *Nothing appears outwardly wrong with the current setup to the eye when viewing the pages or requesting as googlebot (the only issue being the nose dive in organic traffic lol) *both subfolders are setup in apache config file to allow local overrides of .htaccess as follows: <directory store="" widget-group1="">Options -Indexes FollowSymLinks -MultiViews
Technical SEO | | altecdesign
AllowOverride All
Order allow,deny
allow from all</directory> <directory store="">Options -Indexes FollowSymLinks -MultiViews
AllowOverride All
Order allow,deny
allow from all</directory>0 -
How to handle temporary campaign URLs
Hi, We have just run a yearly returning commercial campaign for which we have created optimized URL's. (e.g. www.domain.tld/campaign including the category and brand names after the campaign www.domain.tld./campaign/womens This has resulted in 4500+ URL's being indexed in Google including the campaign name, now the campaign is over and these URL's do not exist anymore. How should we handle those URL's? 1.) 301 them to the correct category without the campaign name 2.) Create a static page www.domain.tld/campaign to which we 301 all URL's that have the campaign name in them Do you have any other suggestions on what the best approach would be? This is a yearly commercial campaign so in a year time we will have the same URL's again. Thanks, Chris
Technical SEO | | eCommerceSEO0 -
Crawling and indexing content
If a page element (div, e.g.) is initially hidden and shown only by a hover descriptor or Javascript call, will Google crawl and index it’s content?
Technical SEO | | Mont0 -
404 Error on Spider Emulators
I recently began working at a company called Uncommon Goods. I ran a few different spider emulators on our homepage (uncommongoods.com) and I saw a 404 Error on SEO-browser.com as well as URL errors on Summit Media's emulator and SEOMoz's crawler. It seems there is a serious problem here. How is this affecting our site from an SEO standpoint? What are the repercussions? Also, I know we have a lot of javascript on our homepage..is this causing the 404? Any advice would be much appreciated. Thanks! -Zack
Technical SEO | | znotes0 -
Handling '?' in URLs.
Adios! (or something), I've noticed in my SEOMoz campaign that I am getting duplicate content warnings for URLs with extensions. For example: /login.php?action=lostpassword /login.php?action=register etc. What is the best way to deal with these type of URLs to avoid duplicate content penelties in search engines? Thanks 🙂
Technical SEO | | craigycraig0