404 from a 404 that 301s
-
I must be missing something or skipping a step or lacking proper levels of caffeine.
Under my High Priority warnings I have a handful of 404s which are like that on purpose but I'm not sure how Moz is finding them. When I check the referrer info, the 404 is being linked to from a different 404 which is now a 301 (due to craziness of our system and what was easiest for the coders to fix a different problem ages ago). Basically, if a user decides to type in a non-existent model number into the URL there is a specific 404 that comes up. While the 404 error is "site.com/product/?model=abc123" the referrer is "site.com/product?model=abc123" (or more simply, one slash is missing). I can't see how Moz is finding the referrer so I can't figure out how to make Moz stop crawling it. I actually have the same problem in Google WMT for the same group of 404s.
What am I just not seeing that will fix this?
-
Let me know if it works Mike. There is actually a third possibility which is;
Some page(s) might generate a dynamic URL only upon being visited by a browser/search agent. If that's the case, then you can set up an event tracking through your website in conjuction with Google Analytics and track teh refferer;
_gaq.push(['_trackEvent', 'Error', '404', 'page: ' + document.location.pathname + document.location.search + ' ref: ' + document.referrer ]);
After you collect some data (Submit your website to Google WMT or wait for next MOZ visit) you can export and run your filter.
The alternative to this method could be one of the 2 following;
- enabling extreme debug/log mode on your programming platform and collect logs for further processing. You can run a small Python script to find the RegEx pattern. I advise to setup a demo copycat of your website on a subdomain and then run this experiment. You can then submit the demo sub domain to Google Webmaster tools and wait for the crawlers.
- Reconfigure your webserver logging (httpd.conf if using Apache) to log more details. Make sure you turn back into to the normal data collecting configuration to avoid storage consumption.
Good luck,
Ali
-
I had done about half of that... I'll take a look at all of it and try again tomorrow following your suggestions and see if I can figure it out then. Thanks.
-
Hi Mike,
Hope all is well. There are two things that might have made this confusion. Either you have some outdated links somewhere on your website that are leading to the custom 404 page or some external link is pointing back to your website with a wrong URL or missing product. In order to find the link (I say so, because a crawler has to hit a link to crawl so there is definitely one), you can use tools like Ahrefs link analysis and see what is pointing where. export to an excel and filter based on a RegEx you'd make out a 404 generating pattern you already have with Moz or Google WMT. You find one and you'll know where they are coming from and how to fix them. You'd be able to write custom redirects in your htaccess if they are not many. If they are many though, htaccess could slow down your website and the best way would be a back-end base redirect either custom coded or through a plugin based on your platform. I would start from
- my error_logs in webserver logs and match them with WMT and Moz report.
- download CSV and import to excel or program of your choice
- filter based on the pattern
- Match it with where you've found the link through Ahref
- and Voila, now you know exactly how to clean them up
Hope this helps Mike,
Have a nice day,
Ali
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How do I fix a 404 redirect chain
How do I fix a 404 redirect chain? I can't seem to find the answer and I'm worried about it effecting my SEO. Any help would be great!
Technical SEO | | sammecooper0 -
404 errors
Hi I am getting these show up in WMT crawl error any help would be very much appreciated | ?escaped_fragment=Meditation-find-peace-within/csso/55991bd90cf2efdf74ec3f60 | 404 | 12/5/15 |
Technical SEO | | ReSEOlve
| | 2 | mobile/?escaped_fragment= | 404 | 10/26/15 |
| | 3 | ?escaped_fragment=Tips-for-a-balanced-lifestyle/csso/1 | 404 | 12/1/15 |
| | 4 | ?escaped_fragment=My-favorite-yoga-spot/csso/5598e2130cf2585ebcde3b9a | 404 | 12/1/15 |
| | 5 | ?escaped_fragment=blog/c19s6 | 404 | 11/29/15 |
| | 6 | ?escaped_fragment=blog/c19s6/Tag/yoga | 404 | 11/30/15 |
| | 7 | ?escaped_fragment=Inhale-exhale-and-once-again/csso/2 | 404 | 11/27/15 |
| | 8 | ?escaped_fragment=classes/covl | 404 | 10/29/15 |
| | 9 | m/?escaped_fragment= | 404 | 10/26/15 |
| | 10 | ?escaped_fragment=blog/c19s6/Page/1 | 404 | 11/30/15 | | |0 -
GWT Soft 404 count is climbing. Important to fix?
In GWT I am seeing my mobile site's soft 404 count slowly rise from 5 two weeks ago to over 100 as of today. If I do nothing I expect it will continue to rise into the thousands. This is due to there being followed links on external sites to thousands of discontinued products we used to offer. The landing page for these links simply says the product is no longer available and gives links to related areas of our site. I know I can address this by returning a 404 for these pages, but doing so will cause these pages to be de-indexed. Since these pages still have utility in redirecting people to related, available products, I want these pages to stay in the index and so I don't want to return a 404. Another way of addressing this is to add more useful content to these pages so that Google no longer classifies them as soft 404. I have images and written content for these pages that I'm not showing right now, but I could show if necessary. But before investing any time in addressing these soft 404s, does anyone know the real consequences of not addressing them? Right now I'm getting 275k pages indexed and historically crawl budget has not been an issue on my site, nor have I seen any anomalous crawl activity since the climb in soft 404s began. Unchecked, the soft 404s could climb to 20,000ish. I'm wondering if I should start expecting effects on the crawl, and also if domain authority takes a hit when there are that many soft 404s being reported. Any information is appreciated.
Technical SEO | | merch_zzounds0 -
Increase 404 errors or 301 redirects?
Hi all, I'm working on an e-commerce site that sells products that may only be available for a certain period of time. Eg. A product may only be selling for 1 year and then be permanently out of stock. When a product goes out of stock, the page is removed from the site regardless of any links it may have gotten over time. I am trying to figure out the best way to handle these permanently out of stock pages. At the moment, the site is set up to return a 404 page for each of these products. There are currently 600 (and increasing) instances of this appearing on Google Webmasters. I have read that too many 404 errors may have a negative impact on your site, and so thought I might 301 redirect these URLs to a more appropriate page. However I've also read that too many 301 redirects may have a negative impact on your site. I foresee this to be an issue several years down the road when the site has thousands of expired products which will result in thousands of 404 errors or 301 redirects depending on which route I take. Which would be the better route? Is there a better solution?
Technical SEO | | Oxfordcomma0 -
404's in WMT are old pages and referrer links no longer linking to them.
Within the last 6 days, Google Webmaster Tools has shown a jump in 404's - around 7000. The 404 pages are from our old browse from an old platform, we no longer use them or link to them. I don't know how Google is finding these pages, when I check the referrer links, they are either 404's themselves or the page exists but the link to the 404 in question is not on the page or in the source code. The sitemap is also often referenced as a referrer but these links are definitely not in our sitemap and haven't been for some time. So it looks to me like the referrer data is outdated. Is that possible? But somehow these pages are still being found, any ideas on how I can diagnose the problem and find out how google is finding them?
Technical SEO | | rock220 -
Locating 404 Page Errors for Deletion
On my SEOmoz report, there are several 404 pages that I assume need deletion. Yes? When I am looking at my pages from the back-end of WordPress, how do I identify these to delete or fix them? In the list of pages I have created, it is not at all apparent when I click into "edit" the page that any of these are broken pages. I think the 404 pages are urls from pages that I changed the url to be more seo friendly, but they don't really exist. I hope this makes sense - it is baffling to me : ) Thank you for any insight and help with getting these cleared. The errors are listed below from the report. Sheryl | 404 : Error http://durangocodentists.com/durango-dentists-why-greg-mann/dentists-in-durango-co/Cosmetic_Dentistry_Services_Teeth_Whitening_Montezuma_CO.html 404 1 0 404 : Error http://durangocodentists.com/durango-dentists-why-greg-mann/dentists-in-durango-co/General_Dentistry_Services_White_Fillings_Montezuma_CO.html 404 1 0 404 : Error http://durangocodentists.com/durango-dentists-why-greg-mann/dentists-in-durango-co/Request_an_Appointment.html 404 1 0 404 : Error http://durangocodentists.com/videos/repairing-teeth/pid%3A4078865 404 1 0 404 : Error http://durangocodentists.com/videos/teeth-whitening/pid%3A4078865 404 1 0 404 : Error http://durangocodentists.com/videos/veneers/pid%3A4078865 | 404 | 1 | 0 |
Technical SEO | | TOMMarketingLtd.0 -
I am using SEOmoz pro software and my blog tags are bringing up 404 errors.
After checking they do bring back a 404 page, so i am wondering what to do. Do i remove all the blog tags? We use a Drupal cms system.
Technical SEO | | AITLtd0 -
Thoughts about stub pages - 200 & noindex ok, or 404?
With large database/template driven websites it is often possible to get a lot of pages with no content on them. What are the current thoughts regarding these pages with no content, options; Return a 200 header code with noindex meta tag Return a 404 page & header code Something else? Thanks
Technical SEO | | slingshot0