Moz was unable to crawl your site on Jun 22, 2020\. We were unable to access your site due to a page timeout on your robots.txt, which prevented us from crawling the rest of your site.
-
Site: www.kpmg.us
Getting robots.txt timeout fail since 02/29/20. We've checked our server logs and see no errors. Went through all the steps of the "Troubleshooter".
Updated robots.txt to allow rogerbot full access:
User-agent: rogerbot
Disallow:Any ideas how to get roger to crawl my site????
-
Back to the "We were unable to access your site due to a page timeout on your robots.txt".
Could it be the sitemap.xml page specified in the robots.txt is too slow?
Sitemap: https://www.kpmg.us/sitemap.xml -
OK. Got a different error: Your site crawl timed out due to a slow server response. Passing this along to IT.
-
We fixed the situation where the robots.txt files download (see: https://www.kpmg.us/robots.txt) but rogerbot still cannot crawl the site due to some "timeout" issue on the robots.txt.
-
Hmmm, seems all our robots.txt files download as text files. But the others (ex: advisory.kpmg.us/robots.txt) work with rogerbot. I've asked our IT folk to see how were serving .txt files.
-
Hi there, thanks for reaching out!
Is the robots.txt for your site located here: "https://www.kpmg.us/robots.txt"?
If so, the issue may be that the robots.txt downloads as a text file which our crawler, rogerbot will be unable to follow. If our crawler is unable to access to the robots.txt it will cause the crawl to fail.
If you're still having issues, please feel free to reach out to the help@moz.com
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz crawling http rather than https site
Our site is secure but when I ask moz to crawl it by giving the root domain including https moz insists on crawling the non secure version. How do i force it to crawl the secure version?
Link Explorer | | media12340 -
Inbound links not found in Moz Pro(though they exist)
These sites have many inbound links but Moz Pro finds no such links. I also used SimilarWebPro Trial to confirm the existing of such links. (these sites have IP block so it requires an VPN to access with Japan IP) osakado.cc
Link Explorer | | Yuji_m
oskd.biz Could anyone figure out why Moz can't find those links to those sites? Any tricks in place to hide those links from SeoMoz? If I pay to upgrade SimilarWebPro but Id rather stay with SeoMoz Pro. thanks in advance.0 -
No backlinks reporting in MOZ OSE?
We have a bunch of links and none of them report in MOZ OSE. Does the MOZ OSE tool have a cache that takes a while to refresh? We have recently moved over to CloudFlare and since the move the numbers do not add up. Here is a link to a ahrefs screenshot. Any insights would be greatly appreciated. moz.png
Link Explorer | | seanallen0070 -
Moz Pro: Linking RDs to Page much lower than Google Search Console
I'm trying to use the Analyze Keyword tool in Moz Pro, and in the SERP Analysis table, my page has a PA of one, and zero root domains linking to it. If I look at the page in Google Search console, it says I have 229 root domains linking to the page from well known domains like github.com, meetup.com, stackoverflow.com, etc. This particular keyword has been tracked in Moz for the last 6 months, but I just noticed that it was extremely low. I am relatively new to Moz, so forgive me if I sound confused, but can someone explain to me how the numbers can be so low?
Link Explorer | | jakebellacera0 -
Top Pages Report References URLs from a Different Domain
I am running a Top Pages Report using Open Site Explorer and it's returning that the top pages for examplesiteA.org are examplesiteB.com. Important to note that examplesiteA.org was a dormant domain for a few years. We recently purchased it and are in the process of reviving it. Has anyone experienced this before? Would this suggest that canonical link elements were put in place prior to it going dormant and that the top pages for this site are now assumed to be on examplesiteB.com?
Link Explorer | | seoaustin0 -
Open Site Explorer - Just Discovered Inbound Links
Hello Moz Forum, Thanks in advance for any help. My website was recently featured in a bunch of high DA journals and magazines. I don't see the inbound links on the open site explorer, but I do see them in recently discovered. My website's Domain Authority hasn't changed a wink. How long does it take, typically to see the recently discovered links under the inbound links portion? Thanks! j1pqVRt
Link Explorer | | FutureSolutionsMedia0 -
Why does Open site explorer only show a fraction of the linked Domains that Google does?
One of my sites ( http://www.georgerossphotography.com ) shows 19 root domains and 155 links on Open Site Explorer. This doesn't pass the sniff test with me and when I look at Google Webmasters they show 120 domains with 1,870 links. I am assuming that Google use it's own data for ranking purposes which makes me question the validity of Open Site Explorer? Am I missing something ? I have been using OSE as my primary tool and now I feel that it has little value. I would appreciate any feedback. George.
Link Explorer | | sirgeorge0 -
Why does the number of the total external links and the followed linking root domains between Open site explorer and the campaign in Moz pro doesn't match?
Opensiteexplorer returns 52.473 total external links and 715 followed linking root domains while my campaign in Moz analytics return 35.899 total external links and 1092 followed linking root domains. Does anyone knows how this is possible?
Link Explorer | | ConversionMobstars0