Moz was unable to crawl your site on Jun 22, 2020\. We were unable to access your site due to a page timeout on your robots.txt, which prevented us from crawling the rest of your site.
-
Site: www.kpmg.us
Getting robots.txt timeout fail since 02/29/20. We've checked our server logs and see no errors. Went through all the steps of the "Troubleshooter".
Updated robots.txt to allow rogerbot full access:
User-agent: rogerbot
Disallow:Any ideas how to get roger to crawl my site????
-
Back to the "We were unable to access your site due to a page timeout on your robots.txt".
Could it be the sitemap.xml page specified in the robots.txt is too slow?
Sitemap: https://www.kpmg.us/sitemap.xml
-
OK. Got a different error: Your site crawl timed out due to a slow server response. Passing this along to IT.
-
We fixed the situation where the robots.txt files download (see: https://www.kpmg.us/robots.txt) but rogerbot still cannot crawl the site due to some "timeout" issue on the robots.txt.
-
Hmmm, seems all our robots.txt files download as text files. But the others (ex: advisory.kpmg.us/robots.txt) work with rogerbot. I've asked our IT folk to see how were serving .txt files.
-
Hi there, thanks for reaching out!
Is the robots.txt for your site located here: "https://www.kpmg.us/robots.txt"?
If so, the issue may be that the robots.txt downloads as a text file which our crawler, rogerbot will be unable to follow. If our crawler is unable to access to the robots.txt it will cause the crawl to fail.
If you're still having issues, please feel free to reach out to the help@moz.com
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why moz is not crawling my Backlinks & Keyword ?
Hello all brother & sisters, I want to know why moz bot crawler are not detected my backlinks & keyword from last 3-4 months not even single one. All other SEO crawler less popular than moz are detecting my keyword & backlinks. May I know is there any option to forcefully moz bot to crawl my all site specially.
Link Explorer | | loknath.tcms1 -
Why would a site (competitor) show up in Keyword SERPs Analysis but not show up in Rank Tracker for the same keyword?
I used Rank Tracker to evaluate a specific site for "geofencing software" - the result was "not in the top 50." However, if I go to Keyword SERP analysis and put in the same keyword "geofencing software," it shows the specific site within the first page results in SERP. I verified the ranking with a google search of my own and the site does show in the first page results. Why is Rank Tracker not matching Keyword SERP results?
Link Explorer | | Bluedot_IO0 -
Bulk Import on Open Site Explorer
Hi guys, Is there a way to upload multiple domains/pages into Open Site Explorer in order to get DA, PA and Spam Score rather than a manual approach?
Link Explorer | | Metropolis0 -
Can improving page authority on all pages help overall domain auth?
Our website is made up of approx 4000 pages - some of which are old pages with a pa of 1. I am removing these pages (ensuring they don't have links etc). Can removing so many low scoring and useless pages impact on domain authority. Our DA jumped 4 points and i wondered if this was helping Example 4000 pages in total 2500 have PA 1 - these get removed will it look better for DA that overall there are now no pages where PA =1 and the site now contains higher PA pages only?
Link Explorer | | acsilver0 -
Does MOZ have a Flash test tool?
I want to test my websites and see if they use Flash, is there a flash check tool like on SEO tool kit here on MOZ? Thanks, Lance
Link Explorer | | BlueprintMM0 -
Why there is high DA for blank brand new sites?
I bought a domain name directly from the registrar 1 year ago. This Domain name has a very good combination of words and .biz extension. I didn't really do anything, I have default wordpress and GA installation. Now I check this site with Moz Bar and OSE and see it has 9DA and 21PA. How come? Quick Facts: Since October last year it has 1000 hits. 9 external links - all links from random domain directories/listings Any ideas how brand new domain name with no original content can have DA9 not DA1?
Link Explorer | | lovemozforever0 -
[No Title] for all Top Pages at Open Site Explorer
49 out of the top 50 pages of the domain www.parallels.com shows [No Title]. I find this to be a major concern. Pages have long been established, title tags haven't changed recently. Am seeing a fair amount of 301 and 404s showing up, but actual OK 200 pages still showing [No Title]. Also seeing some decreases in organic search traffic at Google. Might there be a correlation?
Link Explorer | | ChristianMKG0