Moz was unable to crawl your site on Jun 22, 2020\. We were unable to access your site due to a page timeout on your robots.txt, which prevented us from crawling the rest of your site.
-
Site: www.kpmg.us
Getting robots.txt timeout fail since 02/29/20. We've checked our server logs and see no errors. Went through all the steps of the "Troubleshooter".
Updated robots.txt to allow rogerbot full access:
User-agent: rogerbot
Disallow:Any ideas how to get roger to crawl my site????
-
Back to the "We were unable to access your site due to a page timeout on your robots.txt".
Could it be the sitemap.xml page specified in the robots.txt is too slow?
Sitemap: https://www.kpmg.us/sitemap.xml -
OK. Got a different error: Your site crawl timed out due to a slow server response. Passing this along to IT.
-
We fixed the situation where the robots.txt files download (see: https://www.kpmg.us/robots.txt) but rogerbot still cannot crawl the site due to some "timeout" issue on the robots.txt.
-
Hmmm, seems all our robots.txt files download as text files. But the others (ex: advisory.kpmg.us/robots.txt) work with rogerbot. I've asked our IT folk to see how were serving .txt files.
-
Hi there, thanks for reaching out!
Is the robots.txt for your site located here: "https://www.kpmg.us/robots.txt"?
If so, the issue may be that the robots.txt downloads as a text file which our crawler, rogerbot will be unable to follow. If our crawler is unable to access to the robots.txt it will cause the crawl to fail.
If you're still having issues, please feel free to reach out to the help@moz.com
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Page Authority higher on my old (redirected) domain than new domain?
Hi everyone. I moved the domain my blog was on about 18 months ago, and it's ranking in Google never recovered. I've noticed in the 'Inbound Links' tool that my old domain (jamescrowley.co.uk) shows a page authority of 26, while my new domain (jamescrowley.net) shows a page authority of 15. Any idea why that would be? 301 redirects have been in place for the whole 18 months, so I'd have thought the authority would have been 'passed on' by now? Many thanks James
Link Explorer | | james.crowley0 -
After how long Moz show the matrices of a new domain
I want to know the Matrices of this site " Clash of lights ". Its a new domain how can i find its matrices like DA PA etc in Moz. Please guide me through it.
Link Explorer | | Muhammadahamd0 -
Open site explorer Update
Hi Moz community, I have been using this tool to check my DA & PA and backlinks of my website. Since last week we have some outbound links to my website as I knew it but MOZ havent pointed out these links when I use the "Open site explorer" tool. Please answer this for me. Thanks & regards, Zita
Link Explorer | | zita.vn0 -
Why some of my strong backlinks don't show up in Site Explorer
Hello, Just was checking my backlinks and I see that few very strong backlinks are not showing up in my links list. I check with Moz bar in my browser and those are all dofollow links. Some of those sites link back to me a while ago. Thanks, Regards, Armands
Link Explorer | | A_Fotografy0 -
Open Site Explorer -> Just Discovered -> Internal Links: Only shown 1 url link per day
Hi, We have a page that have like 10-15 new content/pages per day.
Link Explorer | | Inspiringinterns
eg. domain.com/post-id-01/, omain.com/post-id-02/, omain.com/post-id-03/ etc
But somehow, under Just-Discovered, internal links only show 1 per day when it suppose to display like 10-15.
The layout for each pages is exact same except for the content changes and the url's ID changes. Our competitor on the other hand, they have like 5 new page per day, but somehow Moz shows their internal links as 25 per days. Each of their new url have 4 more prefix number.
Eg:
new-page-01?track=990
new-page-01?track=991
new-page-01?track=992 etc
The above example url is technically as 1 page if without the "?" affix. Whiles all our new pages/url have unique id, but Moz only show 1 a day. Please advise on how to get Moz to shows all the new pages in Just-Discovered (internal link). Thanks, Cam0 -
Incorrect crawl errors
A crawl of my websites has indicated that there are some 5XX server errors on my website: Error Code 608: Page not Decodable as Specified Content Encoding
Link Explorer | | LiamMcArthur
Error Code 803: Incomplete HTTP Response Received
Error Code 803: Incomplete HTTP Response Received
Error Code 608: Page not Decodable as Specified Content Encoding
Error Code 902: Network Errors Prevented Crawler from Contacting Server The five pages in question are all in fact perfectly working pages and are returning HTTP 200 codes. Is this a problem with the Moz crawler?1 -
Dofollow link from Moz user profile still not showing in OpenSiteExplorer
Hi everyone! Refering to my last question http://moz.com/community/q/dofollow-link-from-moz-user-profile-not-showing-in-opensiteexplorer, the last update was on March 10 and I'm still searching my dofollow link from my Moz user profile http://moz.com/community/users/625691 after have 200 MozPoints, Opensiteexplorer only found the nofollow link from the icon, but nothing from the anchor text of my URL. Is possible that only take the first link to my blog and ignore the rest if it have the same href? 😞 Thanks! fTdwaO6.png
Link Explorer | | rubenalonsoes1 -
Does the inbound links report include links to all pages of the domain being researched?
If I enter 'abc.com' am I only getting results for 'abc.com' or will I get results for the internal pages of 'abc.com' as well (i.e. 'abc.com/page1.html)? There is a bit of a discrepancy between these results and inbound link results in semrush for example. Then again it seems whenever you use different tools to measure the same thing you get wildly varying results. How do you all deal with that?
Link Explorer | | AISEO0