5xx Crawl Issue might not be issues at all. Help
-
Hi,
I ran a crawl test on our website and it came back with 900 5xx potential errors. When I started opening these links 1 by 1 I could see they were actually working. So i exported the full list of 900 and went to the website: https://httpstatus.io/ pasted the links by 100 and used that. They came back with status codes of 301 / 301 / 200 which i believe means they are okay.
After reading it says that my programmer may need to see if we are blocking the MOZ BOT or to slow the MOZ BOT down. I guess I'm wondering if this is not done is the site actually having these 5xx errors when Google is Crawling or is it just showing 900 errors because of MOZ BOT but actually things are okay?
I know the simple answer is to get the programmer to fix the MOZ BOT issue to know for sure but getting programmers to do things take a lot of time so I'm trying to get a better idea here.
Thanks for your input.
-
Hi there!
Thanks so much for the great question! I'm so sorry to hear you're having this trouble with the 5xx errors. To resolve this we'd recommend adding a crawl delay for rogerbot to your robots.txt file. That crawl delay would look something like this:
User-agent: rogerbot
Crawl-delay: 10This will tell our crawler to slow down when it's crawling. We do not recommend using a crawl delay of longer than 10 as this can keep the crawl from completing.
As far as whether this is impacting Google's ability to crawl, I'm really not able to help identify that. I'm so sorry about that! The best suggestion I can make would be to check the server logs for your site to see how it is responding to other crawlers you may be concerned about.
If you have any other questions about rogerbot or the our tools, please feel free to send an email on over to help@moz.com.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
DA PA issue
Hi, I don't understand why does the DA of my site increasing but not the PA, ay reason guys?
Getting Started | | Seotools530 -
Keyword List issue
Hi, I have created a keyword list with 500 phrases and your software has been set as "Gathering Data" for hours now. I have closed, re-opened, refreshed multiple times and still nothing. Does it really take that long or is there some sort of issue?
Getting Started | | Crowleymjc0 -
My question is, when you translate your website to another language, does moz crawl both or do i have to add another campaign to moz so that they can crawl it seperately?
Hello, i recently translated my website to spanish, keywords,meta tags, content etc. All of our URL for the translated pages start "ep", which symbolizes espanol. My question is, does moz crawl those pages along with my english pages? or do i have add another campaign for moz to crawl my spanish pages seperately?
Getting Started | | prestigeluxuryrentals.com0 -
A lot of duplicate content issues - does Moz understand canonical URL?
Hi, Since I subscribed to Moz my Magento store has given a lot of duplicate content issues. However, I did have a problem with Canonical URL at the time. It has been settled for a couple of weeks by now and although I had 302 redirects before, I configured Magento to 301 today. Since Moz has been crawling and showing duplicate content for exactly the same Magento pages but with endings like store=us, store=aus etc (since I have several store views enabled), I am wondering whether canonical URL does actually help Google to skip these versions of the duplicate pages and does Moz also understand it and will it reduce the amount of duplicate content errors once the 301 redirects and canonical URLs have been properly set for a week or so? Thanks!
Getting Started | | speedbird12290 -
After fixing Crawl Errors, how long does it take to for Moz or Google to re-crawl a website?
Last night I found out through Moz that my robots.txt file was blocking any crawling of my website. I fixed the issue. Now do I just sit and wait?
Getting Started | | cmc-interactive0 -
Is there a way MOZ can help me get HQ links?
I'm new to MOZ, I'm on the niche sites building. Is there an easy way to find HQ pages to post to with MOZ? Like it's with Market samurai.
Getting Started | | bishop230 -
Crawl Diagnostics
Hello Experts, today i was analyse one of my website with moz and get issue overview and get total 212 issue 37 high all derive to this same url http://blogname.blogspot.com/search?updated-max=2013-10-30T17:59:00%2B05:30&max-results=4&reverse-paginate=true so can anyone help me how to find this url and remove all high priority error. and even on page website get A grade then why not performing well in SE ?
Getting Started | | JulieWhite0 -
Is there a way to force re-crawl?
1- Can we force the system to recrawl a website without having to wait for the scheduled date? 2- Please forgive me as I am new to this, but is "Moz Analytics" supposed to replace the "Pro"? Thank you!
Getting Started | | BlackTreeIT0