Moz was unable to crawl your site on Jun 22, 2020\. We were unable to access your site due to a page timeout on your robots.txt, which prevented us from crawling the rest of your site.
-
Site: www.kpmg.us
Getting robots.txt timeout fail since 02/29/20. We've checked our server logs and see no errors. Went through all the steps of the "Troubleshooter".
Updated robots.txt to allow rogerbot full access:
User-agent: rogerbot
Disallow:Any ideas how to get roger to crawl my site????
-
Back to the "We were unable to access your site due to a page timeout on your robots.txt".
Could it be the sitemap.xml page specified in the robots.txt is too slow?
Sitemap: https://www.kpmg.us/sitemap.xml
-
OK. Got a different error: Your site crawl timed out due to a slow server response. Passing this along to IT.
-
We fixed the situation where the robots.txt files download (see: https://www.kpmg.us/robots.txt) but rogerbot still cannot crawl the site due to some "timeout" issue on the robots.txt.
-
Hmmm, seems all our robots.txt files download as text files. But the others (ex: advisory.kpmg.us/robots.txt) work with rogerbot. I've asked our IT folk to see how were serving .txt files.
-
Hi there, thanks for reaching out!
Is the robots.txt for your site located here: "https://www.kpmg.us/robots.txt"?
If so, the issue may be that the robots.txt downloads as a text file which our crawler, rogerbot will be unable to follow. If our crawler is unable to access to the robots.txt it will cause the crawl to fail.
If you're still having issues, please feel free to reach out to the help@moz.com
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Page Authority higher on my old (redirected) domain than new domain?
Hi everyone. I moved the domain my blog was on about 18 months ago, and it's ranking in Google never recovered. I've noticed in the 'Inbound Links' tool that my old domain (jamescrowley.co.uk) shows a page authority of 26, while my new domain (jamescrowley.net) shows a page authority of 15. Any idea why that would be? 301 redirects have been in place for the whole 18 months, so I'd have thought the authority would have been 'passed on' by now? Many thanks James
Link Explorer | | james.crowley0 -
Apart from spying on competitors back link what else can be done in MOZ?
I've recently started trail version in MOZ and I want to explore every possible way that can help me in building my website better. My website Sanctum Consulting (Visas and Immigration)
Link Explorer | | Manifeat90 -
REACT site and sitemap.xml
I have a REACT site www.nettheory.com. MOZ seems to only index the homepage but not all the internally linked pages, even the sitemap.xml is there. The reason I say that is the crawl result only shows the http and https version of the homepage, no other pages mentioned. I also noticed MOZ crawl results point out my content is very thin (<50 words). As a matter of fact, it has a lot more words if the JS runs correctly. Do we know if MOZ crawls REACT or JS based sites correctly?
Link Explorer | | NetTheory-Analysts0 -
Moz Pro: Filter inbound links by partial anchor text?
My site has been targeted by a spam farm with hundreds of different domains, all linking to images on our CDN with similar variations of anchor text, eg: get free high quality hd wallpapers wedding cake makers
Link Explorer | | James_NZ
get free high quality hd wallpapers hairstyle makeover
get free high quality hd wallpapers living room cafe
etc Is it possible within Moz Pro to filter all incoming links with anchor text including "free high quality hd wallpapers" so that I can disavow all of the domains en masse? So far I've only been able to display/download the list of links exactly matching the full anchor text which is very time-consuming with 100+ variations. Regards,
James0 -
Does MOZ have a Flash test tool?
I want to test my websites and see if they use Flash, is there a flash check tool like on SEO tool kit here on MOZ? Thanks, Lance
Link Explorer | | BlueprintMM0 -
MOZ Rank Or MOZ Wrong!?
Hi everyone, I would like to show up you a majestic mistake of ranking. I've checked some of my competitors and encountered that site has no DmT(Domain Trust Rank) but its mT(Moz Trust) is 3.83.Its good! but its not good! which one is true? is it possible that a domain isn't trust but a home page is trust!? zv99ZJV zv99ZJV
Link Explorer | | abolfazli0 -
Open Site Explorer metrics report says we have 9 linking root domains, but way more than 7 are listen. What gives?
When I run a report on my website in OSE (SelectAccount" metrics tells me we have 7 "established" root domains and 9 total links. But the detailed report below has many dozens of links. What is metrics telling me?
Link Explorer | | SelectAccount0 -
Hi guys. My site, www.x-mini.com attained more links and got better alexa ranking. However, my DA and PA dropped. How can I explain this?
My site, www.x-mini.com attained more links and got better alexa ranking. However, my DA and PA dropped. How can I explain this? Siz5lkp.png
Link Explorer | | Dineshr840