Old pages still crawled by SE returning 404s. Better to put 301 or block with robots.txt ?
-
Hello guys,
A client of ours has thousand of pages returning 404 visibile on googl webmaster tools. These are all old pages which don't exist anymore but Google keeps on detecting them. These pages belong to sections of the site which don't exist anymore. They are not linked externally and didn't provide much value even when they existed
What do u suggest us to do:
(a) do nothing
(b) redirect all these URL/folders to the homepage through a 301
(c) block these pages through the robots.txt.
Are we inappropriately using part of the crawling budget set by Search Engines by not doing anything ?
thx
-
Hi Matteo.
The first step I would suggest is determining the source of the links to these 404 pages. If these links are internal to your website, they should be removed or updated.
The next step I would recommend is to ensure your site has a helpful 404 page. The page should offer your site's navigation along with a search function so users can locate relevant content on your site.
I realize that thousands of broken links may seem overwhelming. It is a mess which should be cleaned up. How you proceed is dependent upon how much you value SEO. If your ranking is important and you want to be the best, you will have someone investigate every link and make the appropriate adjustments such as 301 redirecting them to the most appropriate page on your site, or allowing the link to continue to the 404 page.
It's a search engine's job to help users find content. 404s are a natural part of the web. There is nothing inherently wrong with having some 404 pages. Having thousands of pages really shows your site has significant issues. Google's algorithms are not revealed publicly but it's logical to believe they may consider sites with a high percentage of 404 pages less trustworthy. This is my belief but not necessarily that of the SEO community.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Huge httaccess with old 301 redirects. Is it safe to delete all redirects with no traffic in last 2 months?
We have a huge httaccess file over several MB which seems to be the cause for slow server response time. There are lots of 301 redirects related to site migration from 9 months ago where all old URLs were redirected to new URL and also lots of 301 redirects from URL changes accumulated over the last 15 years. Is it safe to delete all 301 redirects which did not receive any traffic in last 2 months ? Or would you apply another criteria for identifying those 301 that can be safely deleted? Any way to get in google analytics or webmaster tools all 301 that received traffic in the last 2 months or any other easy way to identify those, apart from checking the apache log files ?
Intermediate & Advanced SEO | | lcourse0 -
301 old site to new site?
I have client with an old site - www.bestfamilylawattorney.com - which had a lot of spammy links (and bad rankings). Instead of fixing those issues, we started a new URL - www.berenjifamilylaw.com - with new content and redesign. Should I do a 301 redirect from old to new domain? If the old site was being penalized, would a 301 transfer that penalty? I just want to make sure I don't end up hurting the new site after doing all the work to start fresh. Thanks.
Intermediate & Advanced SEO | | mrodriguez14400 -
Redirect old "not found" url (at http) to new corresponding page (now at https)
My least favorite part of SEO 😉 I'm trying to redirect an old url that no longer exists to our new website that is built with https. The old url: http://www.thinworks.com/palm-beach-gardens-team/ New url: https://www.thinworks.com/palm-beach-gardens/ This isn't working with my standard process of the quick redirection plugin in WP or through htaccess because the old site url is at http and not https. Any help would be much appreciated! How do I accomplish this, where do I do it and what's the code I'd use? Thank you Moz community! Ricky
Intermediate & Advanced SEO | | SUCCESSagency0 -
Should pages with rel="canonical" be put in a sitemap?
I am working on an ecommerce site and I am going to add different views to the category pages. The views will all have different urls so I would like to add the rel="canonical" tag to them. Should I still add these pages to the sitemap?
Intermediate & Advanced SEO | | EcommerceSite0 -
Images Returning 404 Error Codes. 301 Redirects?
We're working with a site that has gone through a lot of changes over the years - ownership, complete site redesigns, different platforms, etc. - and we are finding that there are both a lot of pages and individual images that are returning 404 error codes in the Moz crawls. We're doing 301 redirects for the pages, but what would the best course of action be for the images? The images obviously don't exist on the site anymore and are therefore returning the 404 error codes. Should we do a 301 redirect to another similar image that is on the site now or redirect the images to an actual page? Or is there another solution that I'm not considering (besides doing nothing)? We'll go through the site to make sure that there aren't any pages within the site that are still linking to those images, which is probably where the 404 errors are coming from. Based on feedback below it sounds like once we do that, leaving them alone is a good option.
Intermediate & Advanced SEO | | garrettkite0 -
"No index" page still shows in search results and paginated pages shows page 2 in results
I have "no index, follow" on some pages, which I set 2 weeks ago. Today I see one of these pages showing in Google Search Results. I am using rel=next prev on pages, yet Page 2 of a string of pages showed up in results before Page 1. What could be the issue?
Intermediate & Advanced SEO | | khi50 -
Robots.txt unblock
I'm currently having trouble with what appears to be a cached version of robots.txt. I'm being told via errors in my Google sitemap account that I'm denying Googlebot access to the entire site. I uploaded clean and "Allow" robots.txt yesterday, but receive the same error. I've tried "Fetch as Googlebot" on the index and other pages, but still the error. Here is the latest: | Denied by robots.txt |
Intermediate & Advanced SEO | | Elchanan
| 11/9/11 10:56 AM | As I said, there in not blocking on the robots.txt for 24 hours. HELP!0 -
Should I 301 Redirect Old Pages to Newer Ones?
I know there is value having lots of unique content on our websites, but I'm wondering how long it should be kept for, and if there is any value in 301 redirecting it? So, for example we have a number of pages on our website that are dedicated to single products (blue widget x, blue widget y, red widget x, red widget y). Nice unique content, with some (but not many) links. These products are no longer available though and have been replaced. So I'm faced with three choices: 1. Leave it as it is, and hope it adds to the overall site authority (by value of being another page), and also perhaps mop up a few longer tail keywords. Add a link to the replacement product on these pages; 2. 301 redirect these pages to the replacement products to give these a bit of a boost, and lose the content; 3. 301 redirect these pages to the replacement products and move all the old content to a new 'blue widgets archive' and 'red widgets archive' page? Would appreciate everyones thoughts!
Intermediate & Advanced SEO | | BigMiniMan0