Many more 404 being reported in GWT than MA
-
Hi
I have been submitting MA crawl reports to clients developers post going live with a new site migration and instructing them to set up 301's for any 404's still reporting which they have now done (despite instructing them not to go live until all old url's mapped & 301'd to new replacement page, or HP if no replacement).
1) When i look in GWT crawl errors there has been a spike since going live with 660 'page not founds' being reported compared to x11 404's in MA. Could there be a 'lag' in GWT reporting and actually they have already been dealt with just not updated by GWT in this report and the MA report is more accurate ? Should i wait and see/or mark as fixed and see if return tomorrow, or tell dev to immediately investigate ? I have checked some samples links and they are going to 404 type pages so presume they are still broken and urgent issue dev must fix immediately ?
2) How long does it take aprox for the page authority to be transferred via a 301 redirect to a new page since i see some category pages that had good PA and have been 301'd to new category urls, are now showing a PA of just 1 !!
Cheers
Dan
-
Many Thanks !!
-
If the two sites are structured differently and do not share ANY significant content then you don't need the rel alternate href lang tag.
Good luck Dan!
-
Thanks so much for having a look Everett !
I did tell dev to fix all urls (many have been but i see many remain) i.e. not use upp case, remove full stops and spaces, and that redev stage would be good opportunity to fix those along with all the 301 redirects etc. Also further advice received from Tom Roberts about that on this thread too: http://moz.com/community/q/duplicate-title-tags-being-caused-by-upper-case-and-lower-case-version-of-urls All of which i have fwd on to dev.
Client also told to add 100% unique descriptive content at same time as adding products but will remind them.
Re the .com & .co.uk & hreflang - these sites are managed by different teams and content is significantly different i think and site structured quite differently. Would that still require hreflang attribution ?
Thanks again Everett !!
All Best
Dan
-
Hello Dan,
Thank you for following up with the private message. I'll keep the domain name out but will post here so others can learn from it.
I checked the 404 error page and it does return a 404 response in the http header, which is good. However, I did notice a few other issues while on the site...
I think you should block the following directories in your robots.txt file:
/search/
/stockist/
/ukstockist/
/services/form/I also noticed several product pages with absolutely no product detail descriptions (e.g. /happy-tee-pink), which is not a good signal to send to Google, and not a good user experience. You should be creative and write custom, 100% unique content for these pages.
I noticed there is both a .com and .co.uk site with no use of rel alternate href lang. Click here for more information.
You should stop using capitol letters in the URLs. Everything should be lower-case. Also don't use spaces in the URL. Use a dash or underscore instead (e.g. "/Stockist/North East" should be "/stockist/northeast" ).
Google has been executing javascript more and more lately and this is causing a lot of 404 rises, among other errors, across the web. Google Webmaster Tools will tell you exactly which page caused the error and, if applicable, which page was linking to it. This would help diagnose the issue so I suggest looking into that and going from there.
Good luck!
-
Hey Everett
No worries at all really appreciate your input !
I cant share since whilst i know Moz community a highly ethical bunch of lovely people this is a new very precious client of mine i cant afford to risk losing should anyone see and try and pitch for the biz, as unlikely as that is i cant risk the possibility but have pm'd you just in case your happy to have a look.
I started this thread after looking at GWT and looking at the reports. I have told dev to 301 any old pages to their new equivalents or HP if not. Then to apply code as per Altecs advice to to batch process 301's for any remaining. They are increasing at aprox 150 new page not founds each day so something clearly going on but hopefully if dev deploy all the above that should reverse the ever growing spike do u think ?
I have told them once they do that to mark all as fixed in GWT and then just attend to any that recur afterwards on a case by case basis.
Any further help advice much appreciated, especially if above unlikely to resolve ?
All Very Best
Dan
-
Hello Dan,
I'm sorry, but I can't provide more actionable advice without seeing the site. If you're willing to share the site I can have a look at what the URLs are doing. But if you really want to know what's causing the 404 errors, Google Webmaster Tools has a report you can look at, or you can check log files.
-
Thanks Altec
There are only 2x soft 404's being reported
Also im asking them to redirect any 404 to the nearest equivalent page or the home page if no new similar equivalent page. Any others will be dealt with by following your code advice for those occuring due to parameters etc
I take it that if do both above that will avoid any soft 404's ?
Cheers
Dan
-
Dan-
Yep you are right on, if you added the above type code below all of the specific page redirects then that would "catch" all the remaining URL's with random parameters in the URL and redirect them to the home page. And its very similar code for the windows server, I'm confident your Dev should know this, if for some reason they don't I can message you the code snippets you would need for the windows server.
Also Everett Sizemore brought up a great point below. If you are redirecting all 404 pages to this 1 custom "404 landing page" that is incorrect. That is called a "soft 404" error. These will actually show up in your WMT account as well as under a section called "soft 404's".
-
Thanks so much for taking the time to comment Everett, I really appreciate it !!
i just tested that and can confirm entering a nonesense url inc client domain, it is resolving in a 404 page.
Since thats the case can i now ignore this bit:
"If you redirect the user agent to another URL before returning the status code they (e.g. Googlebot) will never know that the original URL they were trying to access has been "removed" and will assume it has just been redirected".
However 404s are growing further every day by the hundreds so does that mean likely as a result of the below?:
"Furthermore, as the parameter following the ?aspxerropath=..." changes each time you will have more and more 404 URLs instead of just the one."
Any further help/advice/clarification re above much appreciated since i ultimately want to take the aspects of this thread that still apply into an email to dev to help them resolve this asap ?
Very Many Thanks
Dan
-
thanks again Altec !
yes my clients just migrated to a new site and typically Google launched Panda 4 straight after so not sure whats causing what at the moment and suppose should wait and see after another week or two before drawing any conclusions.
However page not found errors have spiked and continue to grow at a rate of knots
The clients site is on a windows server so don't think that apaache code would do, dont suppose you know what it would be for windows would you ? or i take it this is something clients developers should know in which case i will simply instruct them to add relevant code ?
I have told dev to redirect ALL old content pages to new equivalent page, so if they add above type code after that then should deal with any remaining non content type page not founds ?
Thanks again Alec !!
-
Hello Dan,
Google is constantly crawling the web and updating their data almost in realtime whereas Moz crawls and updates in batches. The Moz data should catch up after the next refresh, which I think happens about every month.
Something in your example caught my attention. The URL incluided a path indicating a custom 404 page: "Info/FileNotFound.aspx?aspxerrorpath=". Let's say I go to your site and type in some nonesense URL like domain.com/123456.aspx", will I get redirected to this /info/FileNotFound.aspx page? If so, that is not ideal. You want the URL I was attempting to access to return the 404 response so it can be removed from the index. If you redirect the user agent to another URL before returning the status code they (e.g. Googlebot) will never know that the original URL they were trying to access has been "removed" and will assume it has just been redirected. Furthermore, as the parameter following the ?aspxerropath=..." changes each time you will have more and more 404 URLs instead of just the one.
-
No problem Dan.
There is definitely no harm in ticking off the errors as "fixed" and then seeing which ones return. If they return I'm betting 1 of 2 things. 1.) That you still have a live link somewhere on the new site pointing to the page. The easiest way to test this is click on one of the pages in WMT and hit the "linked from" button. That shows you where google is getting sent to the page from. The second option is that the pages have an "external" link pointing too them. If that's the case you want to be double sure to redirect it so you get the link credit!
I'm not 100% positive on why MOZ is not showing the errors as quickly, but MOZ does a "deep" crawl of the web and then does a lot of computing. WMT is literally just crawling your site, and spitting out anything it finds to your account. A much easier job for WMT, hence it is pretty quick to report site errors too you. The errors in your MOZ account will most likely update when MOZ pushes their next mozscape index update (this will also be useful for you because MOZ will have better data about "external" links than WMT)
From the error you sent me it looks like you are going through a move of a site from .aspx to .php or .html. When you change languages like this you can end up with a lot of gnarly 404 pages, especially from search pages that have variables in the URL. If you have a large amount of these errors, so many that you feel you are getting a temporary "penalty" for too many site errors you can write a quick snippet of code in the .httpdconf file on your apache server to fix these old useless pages. (if your not on an apache server this can still be achieved, the code would just be different.)
make sure to put this snippet below all of your other individual page redirects
#redirect all old dead .aspx pages to the new homepage
RedirectMatch 301 /(.*).aspx /You should still make your best effort to redirect each old page to its corresponding "new" page. However at a certain point if you have thousands to redirect, and many are just pages with with search parameters in the URL... there comes a point where you want to just hit all the remaining pages in 1 fell swoop.
I'm not certain on how long it would take the 301 to fully "pass" page rank in Google's eyes. I think it is safe to assume that once the old page no longer has a 404 link, and it is not showing up any longer in Google searches then the PR has been passed. MOZ page authority should reflect it once MOZ pics up the change, but MOZ authority is just very "similar" to PR. Its not an exact science, MOZ should pic up the change easily though, it will just take them a little longer time.
-
Many Thanks Altec !
Any harm in ticking them all off and seeing which return ? Should be more thorough shouldn't it than doing a sample ? and also dev say that many 404's refer to a really old version of the website before they were involved so doubt these are still really there....although the spike only occurred after new site went live so i'm slightly dubious about that response from them.
The list has also grown from x660 to x715 in 24 hours so looks like more being discovered!
Any ideas why Moz Analytics is not discovering & reporting these in crawl error reports (saying only 3x 404s remain compared to GWT saying x715 page not founds) ?
Also what about the kind of pages starting with below, being reported as 404, do they matter ?:
| Info/FileNotFound.aspx?aspxerrorpath= |
Any ideas re my second question, how long does G take to pass authority via a 301 from an old page to its replacement aproximately ? and does Moz Page Authority reflect this in its score once G has passed it ?
Sorry for more questions any feedback much appreciated
Many Thanks
Dan
-
Dan-
It sounds like you have already spot checked a few of the pages and they are still returning errors. In my experience WMT is very quick to re-crawl pages you have marked as fixed to see if they truly are fixed. For example if you mark 100 pages as fixed, then 24 hours later 95 of the pages come back, its always been the case for me that the 95 pages still had an issue and were not "fixed".
I would pull up the error report in WMT again, spot check a few of the examples and see if the pages still error out. If they do I would export that list from WMT, send it to your Dev and ask him to fix all redirection errors.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How can i fix? some information is wrong on the report.
https://moz.com/researchtools/ose/spam-analysis/flags?subdomain=bassinotary.com hi please advice. please see the link to my spam report. some information is wrong. like no email address or social profile link found I have already had links on that page and email and address and schema.org is in place. please help thanks.
Moz Bar | | grbassi0 -
Moz Site Crawl Test 404
Crawled site a number of times using Crawl Test. Its reporting 404's from files that are actually present. What do you make of this? Justin
Moz Bar | | GrouchyKids0 -
Monthly Ranking Comparisons on Custom Reports
Hi, We run MONTHLY Moz custom reports that compares rankings from the previous month, but some of the data just doesn't make sense. As an example, in the previous month’s report our ranking for “keyword X” was #3, when we look at the current month’s report our ranking for “keyword X” is still #3, but it shows that it’s gone up by 3 positions. This position change is not true if you compare to the previous month’s report! How can we know what dates we are comparing against? All the best, Lee.
Moz Bar | | Webpresence0 -
How do I pull a rankings report for a specific week in the past?
The date of the rankings I want is January 13th. I can see the individual keyword history in rank tracker, but I have to go each word. I'd like to just have that report to compare it to, since that is when a lot of our rankings changed. So I need that report to compare our future efforts with. Please advise and thank you! Ruben
Moz Bar | | KempRugeLawGroup0 -
Duplicate Home Page in Moz Report
I recently ran a crawl report in MOZ for www.laddersfree.co.uk and it has picked duplicate pages for my home page. The two pages are www.laddersfree.co.uk and www.laddersfree.co.uk The CMS System my developer is using is Concrete5. If you look at the source code there is a Canonical <link rel="<a class="attribute-value">canonical</a>" href="http://www.laddersfree.co.uk/" /> I am very confused as to why there are two pages picked up in the crawl. Can someone offer any advice please? Thank you. Jason
Moz Bar | | gymmad0 -
Why is the exact same URL being seen as duplicate and showing an error in my SEO reports
Well, I am still having duplicate page issues. I have a question about one of the errors SEO is giving me when I download a crawl report. I am going to attach a screen shot of part of the report so you can see for yourself, along with explaining it here. SEO shows the list of URL's that it crawled in the report. In this(see attachment) portion of the report it has 321 results for the exact same URL. It also says all of these exact same URL's have received a 404 error. What I want to know is how does it make 321 results for the same URL? And with this error that I don't see when I look at the page? 0hkRDST
Moz Bar | | JoshMaxAmps0 -
Signed up for moz reports - have received Moz error report - need someone who is capable to take report and perform cleanup edits within Joomla site?
Looking for someone in the US - please contact me at Mary@workingwebsolutions.com If available and interested in task. Thanks Mary
Moz Bar | | PortlandWebDesign0 -
Moz Report: Number of Domains linking to Domain
In the Moz report, what does the "number of domains linking to domain" indicate? I'm analyzing our competitors link profile and noticed some numbers as high as 2369232? Is there a form that explains the reports a little more in detail? Thanks,
Moz Bar | | WebRiverGroup0