Many more 404 being reported in GWT than MA
-
Hi
I have been submitting MA crawl reports to clients developers post going live with a new site migration and instructing them to set up 301's for any 404's still reporting which they have now done (despite instructing them not to go live until all old url's mapped & 301'd to new replacement page, or HP if no replacement).
1) When i look in GWT crawl errors there has been a spike since going live with 660 'page not founds' being reported compared to x11 404's in MA. Could there be a 'lag' in GWT reporting and actually they have already been dealt with just not updated by GWT in this report and the MA report is more accurate ? Should i wait and see/or mark as fixed and see if return tomorrow, or tell dev to immediately investigate ? I have checked some samples links and they are going to 404 type pages so presume they are still broken and urgent issue dev must fix immediately ?
2) How long does it take aprox for the page authority to be transferred via a 301 redirect to a new page since i see some category pages that had good PA and have been 301'd to new category urls, are now showing a PA of just 1 !!
Cheers
Dan
-
Many Thanks !!
-
If the two sites are structured differently and do not share ANY significant content then you don't need the rel alternate href lang tag.
Good luck Dan!
-
Thanks so much for having a look Everett !
I did tell dev to fix all urls (many have been but i see many remain) i.e. not use upp case, remove full stops and spaces, and that redev stage would be good opportunity to fix those along with all the 301 redirects etc. Also further advice received from Tom Roberts about that on this thread too: http://moz.com/community/q/duplicate-title-tags-being-caused-by-upper-case-and-lower-case-version-of-urls All of which i have fwd on to dev.
Client also told to add 100% unique descriptive content at same time as adding products but will remind them.
Re the .com & .co.uk & hreflang - these sites are managed by different teams and content is significantly different i think and site structured quite differently. Would that still require hreflang attribution ?
Thanks again Everett !!
All Best
Dan
-
Hello Dan,
Thank you for following up with the private message. I'll keep the domain name out but will post here so others can learn from it.
I checked the 404 error page and it does return a 404 response in the http header, which is good. However, I did notice a few other issues while on the site...
I think you should block the following directories in your robots.txt file:
/search/
/stockist/
/ukstockist/
/services/form/I also noticed several product pages with absolutely no product detail descriptions (e.g. /happy-tee-pink), which is not a good signal to send to Google, and not a good user experience. You should be creative and write custom, 100% unique content for these pages.
I noticed there is both a .com and .co.uk site with no use of rel alternate href lang. Click here for more information.
You should stop using capitol letters in the URLs. Everything should be lower-case. Also don't use spaces in the URL. Use a dash or underscore instead (e.g. "/Stockist/North East" should be "/stockist/northeast" ).
Google has been executing javascript more and more lately and this is causing a lot of 404 rises, among other errors, across the web. Google Webmaster Tools will tell you exactly which page caused the error and, if applicable, which page was linking to it. This would help diagnose the issue so I suggest looking into that and going from there.
Good luck!
-
Hey Everett
No worries at all really appreciate your input !
I cant share since whilst i know Moz community a highly ethical bunch of lovely people this is a new very precious client of mine i cant afford to risk losing should anyone see and try and pitch for the biz, as unlikely as that is i cant risk the possibility but have pm'd you just in case your happy to have a look.
I started this thread after looking at GWT and looking at the reports. I have told dev to 301 any old pages to their new equivalents or HP if not. Then to apply code as per Altecs advice to to batch process 301's for any remaining. They are increasing at aprox 150 new page not founds each day so something clearly going on but hopefully if dev deploy all the above that should reverse the ever growing spike do u think ?
I have told them once they do that to mark all as fixed in GWT and then just attend to any that recur afterwards on a case by case basis.
Any further help advice much appreciated, especially if above unlikely to resolve ?
All Very Best
Dan
-
Hello Dan,
I'm sorry, but I can't provide more actionable advice without seeing the site. If you're willing to share the site I can have a look at what the URLs are doing. But if you really want to know what's causing the 404 errors, Google Webmaster Tools has a report you can look at, or you can check log files.
-
Thanks Altec
There are only 2x soft 404's being reported
Also im asking them to redirect any 404 to the nearest equivalent page or the home page if no new similar equivalent page. Any others will be dealt with by following your code advice for those occuring due to parameters etc
I take it that if do both above that will avoid any soft 404's ?
Cheers
Dan
-
Dan-
Yep you are right on, if you added the above type code below all of the specific page redirects then that would "catch" all the remaining URL's with random parameters in the URL and redirect them to the home page. And its very similar code for the windows server, I'm confident your Dev should know this, if for some reason they don't I can message you the code snippets you would need for the windows server.
Also Everett Sizemore brought up a great point below. If you are redirecting all 404 pages to this 1 custom "404 landing page" that is incorrect. That is called a "soft 404" error. These will actually show up in your WMT account as well as under a section called "soft 404's".
-
Thanks so much for taking the time to comment Everett, I really appreciate it !!
i just tested that and can confirm entering a nonesense url inc client domain, it is resolving in a 404 page.
Since thats the case can i now ignore this bit:
"If you redirect the user agent to another URL before returning the status code they (e.g. Googlebot) will never know that the original URL they were trying to access has been "removed" and will assume it has just been redirected".
However 404s are growing further every day by the hundreds so does that mean likely as a result of the below?:
"Furthermore, as the parameter following the ?aspxerropath=..." changes each time you will have more and more 404 URLs instead of just the one."
Any further help/advice/clarification re above much appreciated since i ultimately want to take the aspects of this thread that still apply into an email to dev to help them resolve this asap ?
Very Many Thanks
Dan
-
thanks again Altec !
yes my clients just migrated to a new site and typically Google launched Panda 4 straight after so not sure whats causing what at the moment and suppose should wait and see after another week or two before drawing any conclusions.
However page not found errors have spiked and continue to grow at a rate of knots
The clients site is on a windows server so don't think that apaache code would do, dont suppose you know what it would be for windows would you ? or i take it this is something clients developers should know in which case i will simply instruct them to add relevant code ?
I have told dev to redirect ALL old content pages to new equivalent page, so if they add above type code after that then should deal with any remaining non content type page not founds ?
Thanks again Alec !!
-
Hello Dan,
Google is constantly crawling the web and updating their data almost in realtime whereas Moz crawls and updates in batches. The Moz data should catch up after the next refresh, which I think happens about every month.
Something in your example caught my attention. The URL incluided a path indicating a custom 404 page: "Info/FileNotFound.aspx?aspxerrorpath=". Let's say I go to your site and type in some nonesense URL like domain.com/123456.aspx", will I get redirected to this /info/FileNotFound.aspx page? If so, that is not ideal. You want the URL I was attempting to access to return the 404 response so it can be removed from the index. If you redirect the user agent to another URL before returning the status code they (e.g. Googlebot) will never know that the original URL they were trying to access has been "removed" and will assume it has just been redirected. Furthermore, as the parameter following the ?aspxerropath=..." changes each time you will have more and more 404 URLs instead of just the one.
-
No problem Dan.
There is definitely no harm in ticking off the errors as "fixed" and then seeing which ones return. If they return I'm betting 1 of 2 things. 1.) That you still have a live link somewhere on the new site pointing to the page. The easiest way to test this is click on one of the pages in WMT and hit the "linked from" button. That shows you where google is getting sent to the page from. The second option is that the pages have an "external" link pointing too them. If that's the case you want to be double sure to redirect it so you get the link credit!
I'm not 100% positive on why MOZ is not showing the errors as quickly, but MOZ does a "deep" crawl of the web and then does a lot of computing. WMT is literally just crawling your site, and spitting out anything it finds to your account. A much easier job for WMT, hence it is pretty quick to report site errors too you. The errors in your MOZ account will most likely update when MOZ pushes their next mozscape index update (this will also be useful for you because MOZ will have better data about "external" links than WMT)
From the error you sent me it looks like you are going through a move of a site from .aspx to .php or .html. When you change languages like this you can end up with a lot of gnarly 404 pages, especially from search pages that have variables in the URL. If you have a large amount of these errors, so many that you feel you are getting a temporary "penalty" for too many site errors you can write a quick snippet of code in the .httpdconf file on your apache server to fix these old useless pages. (if your not on an apache server this can still be achieved, the code would just be different.)
make sure to put this snippet below all of your other individual page redirects
#redirect all old dead .aspx pages to the new homepage
RedirectMatch 301 /(.*).aspx /You should still make your best effort to redirect each old page to its corresponding "new" page. However at a certain point if you have thousands to redirect, and many are just pages with with search parameters in the URL... there comes a point where you want to just hit all the remaining pages in 1 fell swoop.
I'm not certain on how long it would take the 301 to fully "pass" page rank in Google's eyes. I think it is safe to assume that once the old page no longer has a 404 link, and it is not showing up any longer in Google searches then the PR has been passed. MOZ page authority should reflect it once MOZ pics up the change, but MOZ authority is just very "similar" to PR. Its not an exact science, MOZ should pic up the change easily though, it will just take them a little longer time.
-
Many Thanks Altec !
Any harm in ticking them all off and seeing which return ? Should be more thorough shouldn't it than doing a sample ? and also dev say that many 404's refer to a really old version of the website before they were involved so doubt these are still really there....although the spike only occurred after new site went live so i'm slightly dubious about that response from them.
The list has also grown from x660 to x715 in 24 hours so looks like more being discovered!
Any ideas why Moz Analytics is not discovering & reporting these in crawl error reports (saying only 3x 404s remain compared to GWT saying x715 page not founds) ?
Also what about the kind of pages starting with below, being reported as 404, do they matter ?:
| Info/FileNotFound.aspx?aspxerrorpath= |
Any ideas re my second question, how long does G take to pass authority via a 301 from an old page to its replacement aproximately ? and does Moz Page Authority reflect this in its score once G has passed it ?
Sorry for more questions any feedback much appreciated
Many Thanks
Dan
-
Dan-
It sounds like you have already spot checked a few of the pages and they are still returning errors. In my experience WMT is very quick to re-crawl pages you have marked as fixed to see if they truly are fixed. For example if you mark 100 pages as fixed, then 24 hours later 95 of the pages come back, its always been the case for me that the 95 pages still had an issue and were not "fixed".
I would pull up the error report in WMT again, spot check a few of the examples and see if the pages still error out. If they do I would export that list from WMT, send it to your Dev and ask him to fix all redirection errors.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Possible bug in Crawl Issues report?
Hi all - My crawl issues report shows 3 pages with missing titles. These are just google verification files and the robot.txt file - shouldn't these be excluded? Pages with Title Missing or Emptyas of May 11
Moz Bar | | A-Drive
URL Page Authority Linking Root Domains
https://www.mysite.com/googlea87e28121c071983.html
1 0
https://www.mysite.com/robots.txt
1 0
https://www.mysite.com/google9b9dc57478f61677.html0 -
Dashboard module in Custom report
Hello Earth I'm putting a custom report together for higher management and have added the dashboard module as a good starting point. However, all I want from it is the overall visit & visit break down and the domain authority data, all the 'Mentions Found' 'On Page Wins' 'Ranking Highlights' info is just making the report over long and less to the point. Is there anyway of taking some of the dashboard into the report instead of the whole kaboodle? Or is there another module that will give me total visits not just organic? Many Thanks in advance Chris
Moz Bar | | ChrisParry0 -
We Launched a new site and Rogerbot is still reporting on links/errors from the old site, is there a way to clear those out?
We are mostly a Branding agency, and have not put a lot of effort into SEO for ourselves... SEO tends to take a backseat to design most of the time, making it a little difficult for me at times when it comes to SEO. We recently launched a new site, http://Roninadv.com/ and the developer and I have done quite a bit of work to make it work well for Google. I was really looking forward to a new crawl report from Roger, but alas, It's like Roger crawled the old site? The new site has been up since last Monday. Is there a way to clear out the old errors? Do I just need to give roger more time?
Moz Bar | | PaulRonin0 -
Correcting a 4xx on my crawl report
How can I correct a 4xx error on my crawl report. This page no longer exists. What can I do?
Moz Bar | | henne0 -
How to get Universal Results to Show in Reports?
This has been a source of frustration for me for a long time. I have asked many times and never found anything in the support documentation on how to add this. Is it something that is simply not offered? If so is it not outrageous to others that this simple feature is not present? What I want: I would like to be able to include Universal Results icons in my rank reports. Why? Well as is the case much of the time my clients Google profile is included in the maps results (thanks to our optimization services) but they are not on the first page. The client can't see the results of this type of SEO. It shows up when logged in but not in the exported reports. How easy would it be to include these? It seems like a no-brainer. If anyone knows where to add them or why they don't automatically include them please let me know. If this feature is not offered I call upon my fellow SEO's to demand this feature. If anyone knows of a way to contact the developers to recommend adding this feature I would also appreciate that information. Thank you Universal Result: Local
Moz Bar | | TotalMarketExposure
You're in it!
Position: before 1st natural result
Contains: 7 results0 -
Where to find one off crawl report
Hello, I don't know if I am being a bit daft but I don't seem to be able to find the area where I can request a one off crawl report anymore (rather than setting up a campaign). Can someone let me know where this is now? Thanks!
Moz Bar | | RikkiD220 -
Report Dates are being displayed in a strange format ??
My reports dates are showing a very strange format (month/day/year) how do we fix to correct format (day/month/year) ? 😉
Moz Bar | | Dan-Lawrence0 -
How can I add Maps Listings Results to reports
I am trying to modify Moz reports so that I can show the maps results but I cant figure out how. Is this a feature that is not offered or am I blind? Even when exporting full rankings to PDF they don't show up. The issue is that a lot of the SEO work we do has a positive effect on the maps rankings for a particular client. Meanwhile the organic rankings are somewhat stagnant for this client and unless I create it manually there isn't anything I can export from Moz that shows maps results inclusion. Any help would be appreciated, thank you. IGlgIYA.jpg
Moz Bar | | TotalMarketExposure1