Moz was unable to crawl your site on Jun 22, 2020\. We were unable to access your site due to a page timeout on your robots.txt, which prevented us from crawling the rest of your site.
-
Site: www.kpmg.us
Getting robots.txt timeout fail since 02/29/20. We've checked our server logs and see no errors. Went through all the steps of the "Troubleshooter".
Updated robots.txt to allow rogerbot full access:
User-agent: rogerbot
Disallow:Any ideas how to get roger to crawl my site????
-
Back to the "We were unable to access your site due to a page timeout on your robots.txt".
Could it be the sitemap.xml page specified in the robots.txt is too slow?
Sitemap: https://www.kpmg.us/sitemap.xml -
OK. Got a different error: Your site crawl timed out due to a slow server response. Passing this along to IT.
-
We fixed the situation where the robots.txt files download (see: https://www.kpmg.us/robots.txt) but rogerbot still cannot crawl the site due to some "timeout" issue on the robots.txt.
-
Hmmm, seems all our robots.txt files download as text files. But the others (ex: advisory.kpmg.us/robots.txt) work with rogerbot. I've asked our IT folk to see how were serving .txt files.
-
Hi there, thanks for reaching out!
Is the robots.txt for your site located here: "https://www.kpmg.us/robots.txt"?
If so, the issue may be that the robots.txt downloads as a text file which our crawler, rogerbot will be unable to follow. If our crawler is unable to access to the robots.txt it will cause the crawl to fail.
If you're still having issues, please feel free to reach out to the help@moz.com
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Looking for someone from Moz to comment on unrealistic spam score
Two years ago I bought domain name aroundtheworldwithme.com as a travel blog. I built the site up slowly and currently have a DA of 28 with decent Google search results. However, according to Moz my spam score is 43%. I am convinced that something funny is going on to give me this spam score. I have gone though all 27 factors that play into the spam score in great detail. I only fail a few of the checks. These are the Double Click Tag, LinkedIn profile, phone number, email, and Facebook Pixel. Which as far as I know, literally zero travel blog websites provide this info. So I am on par with evert other travel blog website. Now I know Moz will say this "doesn't mean your website is spam, just that our algorithm found that websites with similar attributes are spam." But this is completely bogus. All similar websites to mine have spam scores of 1-2%. All other websites I see with spam scores over 40% are literally spam websites. Why am I literally the only legitimate travel blog site that has a spam score over 40%? My backlink profile is similar to all other travel blogs. I actually have less spammy links as most, as I haven't been around too long. So I don't think my backlinks are causing the high spam score. The only thing I can think of is that my domain name used to be owned by someone else. I have a lot of backlinks from a random blog website that were discovered in 2018, three years before I bought the domain. Is it possible that the domain used to be an actual spam site name and I am now being punished for that? If not, then I cannot think of anything that would cause my high spam score other than fundamental defects in the Moz spam score algorithm. Something is going on, and I'd love someone from Moz to actually be able to have a look at my website and tell me why I have such a high spam score. I know Google doesn't care about Moz's spam score (thankfully) but other websites don't want links from me due to my completely bogus spam score. Thanks everyone
Link Explorer | | Heckmantis
aroundtheworldwithme.com0 -
When will Dotbot see changes to my robots.txt file?
Dotbot doesn't want to know anything about my site, my Page Authotize is still 1 and my DA is the same. How long does it take to start getting results? In all the websites we have been working on, we have no problems, but in ours it has come up.
Link Explorer | | Expansyon1 -
Can I change the Link Explorer from check with Google US?
I'm trying to check the ranking keywords for a specific article with the Link Explorer, but when I hover over Ranking Keywords on the Overview it tells me that it's looking on Google US. We're a UK based company and the keywords we're trying to rank for are UK based - is there a way of changing this to match?
Link Explorer | | Kestra0 -
Can MOZ keyword explorer output sentences instead keywords?
Can MOZ keyword explorer output sentences instead keywords? We need to optimize our website for the Knowledge Graph (OG). There was an old tool that does it, but I can't remember its name.
Link Explorer | | Elchanan1 -
Page Authority different between Moz Open Site Explorer and On Page analyser
I have a client who has an old URL with 3 linking root domains and 4 links, with a PA of 24. This is on the open site explorer. There is a 301 redirect in place to direct this old url to the new url. When he uses the Moz bar on page, it shows that he has a PA of only 1 with no LRDs or inbound links? Can anyone please explain why this is happening and if this could affect further page links across his site? It would be appreciated. Kind regards
Link Explorer | | lisa_rothery0 -
How many pages Google crawl in free version
I am using moz pro 30 days trial version.Can Anybody tell me how many pages moz crawl in a day or in a week.Because its two days and only 2 pages they crawled. Thanks
Link Explorer | | VarinderS0 -
.Org are not appearing on the open site explorer
I am new to backlinking and I am learning as part of my role. My question is do .orgs not appear on the open site explorer? This is one for a local website. http://www.pinckneyschools.org/index.php/partners Thanks, Roger Castillo http://www.pinckneyschools.org/index.php/partners
Link Explorer | | LowrySolutions0 -
What could have caused Total links to drop from 860,000 to 93,127 in Open Site Explorer?
I've been monitoring root domain total links for a website over the past couple of months and recently noticed a significant drop at the beginning of July.Specifically, I have been tracking the total links for the root domain using open site explorer. Since April the total number or links has been fluctuating among 860,000 - 865,000. However at the beginning of July, they dropped to 93,127 and now sit only at 110,831.This is a very significant and troubling decrease and I am wondering what the cause could be? Did Moz change the way they report/determine the total link metrics? It is especially baffling since neither total internal nor total external links have seen significant decreases.
Link Explorer | | sixspokemedia0