Moz was unable to crawl your site on Jun 22, 2020\. We were unable to access your site due to a page timeout on your robots.txt, which prevented us from crawling the rest of your site.
-
Site: www.kpmg.us
Getting robots.txt timeout fail since 02/29/20. We've checked our server logs and see no errors. Went through all the steps of the "Troubleshooter".
Updated robots.txt to allow rogerbot full access:
User-agent: rogerbot
Disallow:Any ideas how to get roger to crawl my site????
-
Back to the "We were unable to access your site due to a page timeout on your robots.txt".
Could it be the sitemap.xml page specified in the robots.txt is too slow?
Sitemap: https://www.kpmg.us/sitemap.xml -
OK. Got a different error: Your site crawl timed out due to a slow server response. Passing this along to IT.
-
We fixed the situation where the robots.txt files download (see: https://www.kpmg.us/robots.txt) but rogerbot still cannot crawl the site due to some "timeout" issue on the robots.txt.
-
Hmmm, seems all our robots.txt files download as text files. But the others (ex: advisory.kpmg.us/robots.txt) work with rogerbot. I've asked our IT folk to see how were serving .txt files.
-
Hi there, thanks for reaching out!
Is the robots.txt for your site located here: "https://www.kpmg.us/robots.txt"?
If so, the issue may be that the robots.txt downloads as a text file which our crawler, rogerbot will be unable to follow. If our crawler is unable to access to the robots.txt it will cause the crawl to fail.
If you're still having issues, please feel free to reach out to the help@moz.com
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Issue with former site redirect
Hello, After 2 months of starting a new service with a new domain name and a rather flat curve, I decided to invest in auditing tools. To my surprise, when analyzing links with 'Link Explorer', I got the following warning : "You entered the URL crapahute.com which redirects to www.krapahute.com. Click here to analyze www.krapahute.com instead."... So I understand I bought a domain name that was previously in use, 8 years ago. And logically, when they moved from crapahute.com to krapahute.com, the former site owners set a (probably 401) redirect between their former name and their new one. Probably that in the last 8 years they dropped crapahute.com since I was able to buy the domain name last year. I can't understand the warning I'm receiving. since I thought this redirect would become inactive when they dropped the domain name. On the contrary, if I analyze the backlinks to krapahute.com I can indeed see : crapahute.com [no anchor text] 3 2 0 --
Link Explorer | | x_all
redirect More Info
Date first seen - Date lost - Link target
2016-09-05 - N/A - www.krapahute.com So my question is : where is this warning coming from ? I performed some nslookup and dig on my domain and their's just to find out that every thing seems to look normal. Meanwhile, if I type "crapahute" in Google (and most other search engines), my site arrives in 12th page (!), and if I type "crapahute.com", the site "krapahute.com" appears first. Thank you for those reading up to that point ! Any hint on what could cause this ? Should I change of domain name ? Thank you !0 -
MOZ DA IS NOT INCRAESING
Hey my website on business cards its DA is not increasing. is there any way to increase its DA.
Link Explorer | | cartwrightkar0 -
Ranking Keywords are not showing on Moz
i just check my Apk Finds on moz site explorer and check its ranking keywords but it does not show it please help me my site age is 3 months
Link Explorer | | malik12378670 -
Open site explorer Update
Hi Moz community, I have been using this tool to check my DA & PA and backlinks of my website. Since last week we have some outbound links to my website as I knew it but MOZ havent pointed out these links when I use the "Open site explorer" tool. Please answer this for me. Thanks & regards, Zita
Link Explorer | | zita.vn0 -
How long will it take for the changes we've made to reflect in Moz OSE spam score data?
I signed up for Moz to see the spam flags our site had triggered. As soon as I found out, we worked on it and have been trying to correct our mistakes but it's been more than a month and we've managed to neutralise zero flags. I would appreciate if someone can clarify how long the OSE data takes to refresh. Also, how do you combat the following three specific flags: Ratio of Followed to Nofollowed Subdomains Ratio of Followed to Nofollowed Domains Low Number of Pages Found Crawl only gets a valid response to a small number of pages. Thanks.
Link Explorer | | Oziti0 -
Link Acquisition Assistant redirect to Open Site Explorer??????
Link Acquisition Assistant redirect to Open Site Explorer?????? How can i use Link Acquisition Assistant?
Link Explorer | | bondhoward0 -
Moz crawler showing pages blocked by robots.txt
I've blocked a large number of pages which Moz were showing as duplicate or giving 404's in our robots.txt using /?key and /?p etc. However Moz crawler is still showing as being an issue. I assumed Roger picked up the robots.txt file, or is that not the case?
Link Explorer | | ahyde0 -
Moz crawling bot
Hi guys, in OpenSiteExplorer -> Top Pages, there are no page titles displayed in a raport for certain domain, and "HTTP Status" column shows: "Blocked by robots.txt". I tried to find out what the ID of Moz crawling bot is, and on this page: http://moz.com/community/q/seomoz-spider-bot-details someone says it's: Mozilla/5.0 (compatible; rogerBot/1.0; http://www.seomoz.org/dp/rogerbot). However, my robots.txt doesn't have such entry. Take a look: Automatically banned scanners and crawlers section User-agent: 008 Disallow: / user-agent: AhrefsBot Disallow: / User-agent: MJ12bot Disallow: / User-agent: metajobbot Disallow: / User-agent: Exabot Disallow: / User-agent: Ezooms Disallow: / User-agent: fyberspider Disallow: / User-agent: dotbot Disallow: / User-agent: MojeekBot Disallow: / Section end What could be the problem here, then? Why does the Moz bot think I'm blocking it?
Link Explorer | | superseopl0