Moz was unable to crawl your site on Jun 22, 2020\. We were unable to access your site due to a page timeout on your robots.txt, which prevented us from crawling the rest of your site.
-
Site: www.kpmg.us
Getting robots.txt timeout fail since 02/29/20. We've checked our server logs and see no errors. Went through all the steps of the "Troubleshooter".
Updated robots.txt to allow rogerbot full access:
User-agent: rogerbot
Disallow:Any ideas how to get roger to crawl my site????
-
Back to the "We were unable to access your site due to a page timeout on your robots.txt".
Could it be the sitemap.xml page specified in the robots.txt is too slow?
Sitemap: https://www.kpmg.us/sitemap.xml
-
OK. Got a different error: Your site crawl timed out due to a slow server response. Passing this along to IT.
-
We fixed the situation where the robots.txt files download (see: https://www.kpmg.us/robots.txt) but rogerbot still cannot crawl the site due to some "timeout" issue on the robots.txt.
-
Hmmm, seems all our robots.txt files download as text files. But the others (ex: advisory.kpmg.us/robots.txt) work with rogerbot. I've asked our IT folk to see how were serving .txt files.
-
Hi there, thanks for reaching out!
Is the robots.txt for your site located here: "https://www.kpmg.us/robots.txt"?
If so, the issue may be that the robots.txt downloads as a text file which our crawler, rogerbot will be unable to follow. If our crawler is unable to access to the robots.txt it will cause the crawl to fail.
If you're still having issues, please feel free to reach out to the help@moz.com
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why moz and ahrefs reported a remove link that is exist?
Hi in last few days I checked my website in moz link explorer and ahrefs and it showed lots of my backlink as removed! But when I checked this on the browser its OK is there any problem with these tools? Anyone has this problem? Thanks
Link Explorer | | namibiagonzo0 -
Decrease in page Authority
My Page Authority Has been decrease from 31 to 24 in a week while my DA has increased from 38 to 39.I have checked all my latest link but didn't found any spamy links.Please help me .Whether this is a update or something else
Link Explorer | | sapna19910 -
Page Authority different between Moz Open Site Explorer and On Page analyser
I have a client who has an old URL with 3 linking root domains and 4 links, with a PA of 24. This is on the open site explorer. There is a 301 redirect in place to direct this old url to the new url. When he uses the Moz bar on page, it shows that he has a PA of only 1 with no LRDs or inbound links? Can anyone please explain why this is happening and if this could affect further page links across his site? It would be appreciated. Kind regards
Link Explorer | | lisa_rothery0 -
Site moved to https and not showing any Moz DA or PA
Hello Fellow Mozzers, So, I have a client site that we moved from non https to https at the end of 2015. We had decent moz numbers (Da/Pa etc) but now we are only showing a 1 for both of them. Does it usually take time for Moz to carry over the info to the https version or does that process entirely start over? Not sure where to start looking. Cheers D
Link Explorer | | DarinPirkey0 -
How to force moz to crawl my backlinks?
I have some good number number of backlinks in my webmaster tools. But, open site explorer is showing very few backlinks. How to force moz to crawl all the backlinks? Or is there any way to submit backlinks to moz?
Link Explorer | | sankar7890 -
Why MoZ Bar gives different PA and DA to my homepage? Surly they should have the same metrics?
Hi guys, is it normal that when looking at my homepage with the mozbar i get the same DA and PA? It seems that it should provide the same results. Thankss
Link Explorer | | artdivision0 -
The difference between the statistics MOZ & Ahrefs
Hello dear. I have a website for a year. I made 10K links in one year(from 45 Root Domains).the Ahrefs.com has show full list of link root domains and pages. the ahrefs rank is 49. but MOZ said my Established Links are 5! So I have some questions, please help me: 1- what is difference between MOZ rank and ahrefs rank? 2-When Just-Discovered Link Calculate for my site? 3-why my moz trust is 0.00!?How to improve it? URL: https://esta-register.org
Link Explorer | | allyunit0 -
Discrepancy between number of links from web master tools and open site explorer.
Hi I am looking at number of link to www.bagnodesignglasgow.com on open site explorer shows as total links 24 but on WMT it says 937 ? Why are these two numbers so different ? Thanks
Link Explorer | | VIVOWeb0