Rogerbot does not catch all existing 4XX Errors
-
Hi I experienced that Rogerbot after a new Crawl presents me new 4XX Errors, so why doesn't he tell me all at once?
I have a small static site and had 9 crawls ago 10 4XX Errors, so I tried to fix them all.
The next crawl Rogerbot fount still 5 Errors so I thought that I did not fix them all... but this happened now many times so that I checked before the latest crawl if I really fixed all the errors 101%.Today, although I really corrected 5 Errors, Rogerbot digs out 2 "new" Errors. So does Rogerbot not catch all the errors that have been on my site many weeks before?
Pls see the screenshot how I was chasing the errors
-
I understand,
I am not using a CMS and the site is not very big, so I wondered why Roberbot did not find all the 404 Error at the first time, because they have been there for many months.
Holger
-
Hey Holger,
Our crawler will catch as many errors as it can. It's possible that these errors were not present or just were not found at the time of the crawl.I'm running a crawl test to see if there's any discrepancy between your current campaign crawl and mine just to double-check.
In general, Kyle is correct that sometimes those errors just crop up, especially if you're using any sort of CMS.
I hope that helps. I'll update here after my crawl test is done.
Cheers,
Joel. -
Hi Holger,
4XX Errors can be quite common depending on your site setup so don't be surprised that Roger will keep returning errors for you to fix.
I would advise checking this data against GWT's own crawl error data which you can find in Webmaster Tools under Health>Crawl Errors.
I hope that helps,
K
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Error Code 804: HTTPS (SSL) Error Encountered
I'm seeing the following error in Moz as below. I have not seen any errors when crawling with different tools - is this common or is it something that needs to be looked at. SO far I only have info below. Assuming I would need to open a ticket with hosting provider for this? Thanks! Error Code 804: HTTPS (SSL) Error Encountered Your page requires an SSL security certificate to load (using HTTPS), but the Moz Crawler encountered an error when trying to load the certificate. Our crawler is pretty standard, so it's likely that other browsers and crawlers may also encounter this error. If you have this error on your homepage, it prevents the Moz crawler (and some search engines) from crawling the rest of your site.
Moz Pro | | w4rdy0 -
Errors in my coding how significant is this regarding rankings ?
I posted a question on here yesterday about the homepage asking for advice regarding the content and then was told by two people were very helpful bbut moved over comment not on content but to say taht the major problem was that the coding on my website basically has too many errors which would result in me receiving lower rankings in the search engines. I realise this website is old-fashioned Dreamweaver template which was constructed several years ago which I've updated and I'm certainly not a professional, but I watch my Google analytics and there doesn't seem to be any significant change in the stats from this time last year. This is the site http://www.endeavourcottage.co.uk/ I realise the site is old format and has been around for several years it's just from customer feedback they seem to think it looks okay for the products old cottages but I guess technically it's not the best now. I have run a test using Silktide Nibbler - a free online service that gives you a good complete overview of your website with an overall score. And it did give my website an overall good score but did point out errors in the coding but when I checked some of my competitors near the top of Google for the short tail keywords some of them also have errors in their coding, very similar to my own error score.. I then went to Google Webmaster tools and there were no warning messages. So the big question is how important are these errors scores when it appears that most of the top competition also are in the same situation? I think it's quite possible I could do with a redesign using responsive design Best Alan
Moz Pro | | WhitbyHolidayCottages0 -
404 : Errors in crawl report - all pages are listed with index.html on a WordPress site
Hi Mozers, I have recently submitted a website using moz, which has pulled up a second version of every page on the WordPress site as a 404 error with index.html at the end of the URL. e.g Live page URL - http://www.autostemtechnology.com/applications/civil-blasting/ Report page URL - http://www.autostemtechnology.com/applications/civil-blasting/index.html The permalink structure is set as /%postname%/ For some reason the report has listed every page with index.html at the end of the page URL. I have tried a number of redirects in the .htaccess file but doesn't seem to work. Any suggestions will be strongly appreciated. Thanks
Moz Pro | | AmanziDigital0 -
Having 1 page crawl error on 2 sites
Help! A few weeks back, my dev team did some "changes" (that I don't know anything about), but ever since then, my Moz crawl has only shown one page for either http://betamerica.com or http://fanex.com. Moz service was helpful in talking about a redirect loop that existed, and I asked my team to fix it, which it looks to me like they have. Still, 1 page. I used SEO Book's spider tool and it also only sees 1 page, and sees the sites as http://https://betamerica.com (for example), which is just weird. I don't know enough about HT Access or server stuff to figure out what's going on, so if someone can help me figure that out, I'd appreciate it.
Moz Pro | | BetAmerica0 -
Perplexed - Errors increasing, moz rank dropping, conflicting data from other sources. Please Help.
Hi there, I'm in a bit of a bind and could seriously do with some help. For the past month I've been working with a client to resolve errors onsite. In that period my moz rank has plummeted and my errors (dupe title & content) have increased dramatically. The correlation makes sense, the increase - not so much. Here's why, screaming frog is reporting that the majority of duplicate titles have been removed or dealt with. I've also got the assurance of the developer that the suggested changes are being implemented. Unfortunately, I'm not familiar with the CMS - umbraco - and this is the first time I'm working with this particular developer, so I'm not sure how to gauge progress without using moz tools. So, here are my immediate questions: 1. Why would I get different results from Screaming Frog and Rogerbot? 2. Has anyone here been in a similar situation and could they recommend areas to watch for errors creeping in? 3. Is there a way of identifying which errors have occurred when? Narrowing it down to the week each error occurred would be good enough. 4. Is anybody aware of any inherent SEO flaws in umbraco or common SEO mistakes made using the CMS? 5. is there anything I could provide you with to help you help me? 6. Do you have a suggestion? I'm pretty stuck. Thanks in advance.
Moz Pro | | KJDMedia0 -
Title Missing or Empty 6000+ errors
This seems one issue that can affect my curent rank of site. I own http://www.funnygusta.com and i saw yesterday a drop in traffic from 6000uniques to 3500. Yesterday also i received a new crawl report from seomoz. There was like more than 6000 title missing or empty errors. This have a direct corelation between rank and these errors? Also its weird that i use title on all posts also i use seo plugin by yoast and titles are automatic filled. Any ideea from where to begin? thanks
Moz Pro | | xplicit0 -
How to get rogerbot whitelisted for application firewalls.
we have recently installed an application firewall that is blocking rogerbot from crawling our site. Our IT department has asked for an IP address or range of IP addresses to add to the acceptable crawlers. If rogerbot has a dynamic IP address how to we get him added to our whitelist? The product IT is using is from F5 called Application Security Manager.
Moz Pro | | Shawn_Huber0 -
Crawl Diagnostic Errors
Hi there, Seeing a large number of errors in the SEOMOZ Pro crawl results. The 404 errors are for pages that look like this: http://www.example.com/2010/07/blogpost/http:%2F%2Fwww.example.com%2F2010%2F07%2Fblogpost%2F I know that t%2F represents the two slashes, but I'm not sure why these addresses are being crawled. The site is a wordpress site. Anyone seen anything like this?
Moz Pro | | rosstaylor0