Moz was unable to crawl your site on Jun 22, 2020\. We were unable to access your site due to a page timeout on your robots.txt, which prevented us from crawling the rest of your site.
-
Site: www.kpmg.us
Getting robots.txt timeout fail since 02/29/20. We've checked our server logs and see no errors. Went through all the steps of the "Troubleshooter".
Updated robots.txt to allow rogerbot full access:
User-agent: rogerbot
Disallow:Any ideas how to get roger to crawl my site????
-
Back to the "We were unable to access your site due to a page timeout on your robots.txt".
Could it be the sitemap.xml page specified in the robots.txt is too slow?
Sitemap: https://www.kpmg.us/sitemap.xml
-
OK. Got a different error: Your site crawl timed out due to a slow server response. Passing this along to IT.
-
We fixed the situation where the robots.txt files download (see: https://www.kpmg.us/robots.txt) but rogerbot still cannot crawl the site due to some "timeout" issue on the robots.txt.
-
Hmmm, seems all our robots.txt files download as text files. But the others (ex: advisory.kpmg.us/robots.txt) work with rogerbot. I've asked our IT folk to see how were serving .txt files.
-
Hi there, thanks for reaching out!
Is the robots.txt for your site located here: "https://www.kpmg.us/robots.txt"?
If so, the issue may be that the robots.txt downloads as a text file which our crawler, rogerbot will be unable to follow. If our crawler is unable to access to the robots.txt it will cause the crawl to fail.
If you're still having issues, please feel free to reach out to the help@moz.com
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why some of my backlinks are not showing on moz
why some of my backlinks are not showing on moz. These backlinks are profile and web 2.0 backlinks, i have tried many solutions but none of them worked for me. my site is solution4people is there any way to show all the backlinks of my site on moz . Thanks alot.
Link Explorer | | tzoubaier0 -
Drop in Domain Authority across 20 sites
Hi I monitor and track 20-25 sites. I got a -1 in DA across the board this month. Has anyone else experienced this. It's possible my server/cms guys have made a global change. Has anyone at Moz or anyone in the community experienced this?
Link Explorer | | Andrew-SEO0 -
Open Site Explore page titles show as "No Title"
This has been asked many times but I cannot find an answer. On Open Site Explorer, most URLs I enter show as "No title" which means you have to hover over the URL to see which page is being referred to. I know you'd want an example, so here's your own site Moz.com 🙂 CWrkxZQ
Link Explorer | | clifra0 -
Dofollow link from Moz user profile still not showing in OpenSiteExplorer
Hi everyone! Refering to my last question http://moz.com/community/q/dofollow-link-from-moz-user-profile-not-showing-in-opensiteexplorer, the last update was on March 10 and I'm still searching my dofollow link from my Moz user profile http://moz.com/community/users/625691 after have 200 MozPoints, Opensiteexplorer only found the nofollow link from the icon, but nothing from the anchor text of my URL. Is possible that only take the first link to my blog and ignore the rest if it have the same href? 😞 Thanks! fTdwaO6.png
Link Explorer | | rubenalonsoes1 -
Error message coming up for Open Site Explorer
When in Open Site Explorer there seems to be an error getting the data for http://vagabondtoursofireland.ie/ or www.vagabondtoursofireland.ie I have used this with other websites and have never had a problem. Thanks.
Link Explorer | | Johnny_AppleSeed0 -
Question in regards to Comparing Metrics with MOZ tools
Hi all! After using the "Compare Links Metrics" in the research tools, a few questions popped up. Particularly, in regards to what key metrics to look for when determining if a websites outbound links are trustworthy and may pass Link-juice. Basically, "if my link is put on that website, will search engines trust it"? I've copied an example below and typed a few comments next to particular metrics. Please share some insight them: Page Authority: 48 Page MozRank: 4.94 -Is 4.94 considered OK? Page MozTrust: 4.96 -Is 4.96 considered OK? Internal Equity-Passing Links: 2,819 External Equity-Passing Links: 2,326 -Lower = better? Total Equity-Passing Links: 5,145 Total Internal Links: 2,844 Total External Links: 2,383 - Lower = better? Total Links: 5,227 Followed Linking Root Domains: 88 Total Linking Root Domains: 129 Linking C Blocks: 30 Equity-Passing Links vs
Link Explorer | | 90miLLA
Non-Equity-Passing Links:** I am assuming that the higher the Non-equity passing links VS the Equity passing links is typically better? What would be an idea ratio?** Internal Links vs
External Links: Lower external = better?0 -
Open Site Explorer and Expired Links?
Hi guys, I've been checking a few domains for SEO purposes and it appears that OSE is indexing a lot of links which no longer exist, this has been the case for over 6 months - does OSE cleanse expired links? As I'm sure they will be contributing to authority in the eyes of Moz?
Link Explorer | | lotterybase0 -
Facebook "likes" not showing up in Open Site Explorer
Hi All: I'm dealing with a company that has 299 Facebook likes, but Open Site Explorer only shows three. I checked the NAP on Facebook and it's correct. Anyone know why this might be happening? I'm worried if Open Site Explorer is not seeing the likes, Google might not as well, for whatever reason. Thanks! Wes
Link Explorer | | wrconard0