Moz was unable to crawl your site on Jun 22, 2020\. We were unable to access your site due to a page timeout on your robots.txt, which prevented us from crawling the rest of your site.
-
Site: www.kpmg.us
Getting robots.txt timeout fail since 02/29/20. We've checked our server logs and see no errors. Went through all the steps of the "Troubleshooter".
Updated robots.txt to allow rogerbot full access:
User-agent: rogerbot
Disallow:Any ideas how to get roger to crawl my site????
-
Back to the "We were unable to access your site due to a page timeout on your robots.txt".
Could it be the sitemap.xml page specified in the robots.txt is too slow?
Sitemap: https://www.kpmg.us/sitemap.xml -
OK. Got a different error: Your site crawl timed out due to a slow server response. Passing this along to IT.
-
We fixed the situation where the robots.txt files download (see: https://www.kpmg.us/robots.txt) but rogerbot still cannot crawl the site due to some "timeout" issue on the robots.txt.
-
Hmmm, seems all our robots.txt files download as text files. But the others (ex: advisory.kpmg.us/robots.txt) work with rogerbot. I've asked our IT folk to see how were serving .txt files.
-
Hi there, thanks for reaching out!
Is the robots.txt for your site located here: "https://www.kpmg.us/robots.txt"?
If so, the issue may be that the robots.txt downloads as a text file which our crawler, rogerbot will be unable to follow. If our crawler is unable to access to the robots.txt it will cause the crawl to fail.
If you're still having issues, please feel free to reach out to the help@moz.com
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Looking for someone from Moz to comment on unrealistic spam score
Two years ago I bought domain name aroundtheworldwithme.com as a travel blog. I built the site up slowly and currently have a DA of 28 with decent Google search results. However, according to Moz my spam score is 43%. I am convinced that something funny is going on to give me this spam score. I have gone though all 27 factors that play into the spam score in great detail. I only fail a few of the checks. These are the Double Click Tag, LinkedIn profile, phone number, email, and Facebook Pixel. Which as far as I know, literally zero travel blog websites provide this info. So I am on par with evert other travel blog website. Now I know Moz will say this "doesn't mean your website is spam, just that our algorithm found that websites with similar attributes are spam." But this is completely bogus. All similar websites to mine have spam scores of 1-2%. All other websites I see with spam scores over 40% are literally spam websites. Why am I literally the only legitimate travel blog site that has a spam score over 40%? My backlink profile is similar to all other travel blogs. I actually have less spammy links as most, as I haven't been around too long. So I don't think my backlinks are causing the high spam score. The only thing I can think of is that my domain name used to be owned by someone else. I have a lot of backlinks from a random blog website that were discovered in 2018, three years before I bought the domain. Is it possible that the domain used to be an actual spam site name and I am now being punished for that? If not, then I cannot think of anything that would cause my high spam score other than fundamental defects in the Moz spam score algorithm. Something is going on, and I'd love someone from Moz to actually be able to have a look at my website and tell me why I have such a high spam score. I know Google doesn't care about Moz's spam score (thankfully) but other websites don't want links from me due to my completely bogus spam score. Thanks everyone
Link Explorer | | Heckmantis
aroundtheworldwithme.com0 -
Open Site Explorer Showing Blank Screen
Hi Moz Community, When I go to search for backlinks for any site, I am getting a blank screen after starting the search. This is happening to every search I perform for any URL or root domain. It's been happening for about a week now and I am not able to find any solutions. Any help would be greatly appreciated. Thanks! Ryan
Link Explorer | | RyanRhodesGW0 -
Page Authority different between Moz Open Site Explorer and On Page analyser
I have a client who has an old URL with 3 linking root domains and 4 links, with a PA of 24. This is on the open site explorer. There is a 301 redirect in place to direct this old url to the new url. When he uses the Moz bar on page, it shows that he has a PA of only 1 with no LRDs or inbound links? Can anyone please explain why this is happening and if this could affect further page links across his site? It would be appreciated. Kind regards
Link Explorer | | lisa_rothery0 -
Open Site Explore page titles show as "No Title"
This has been asked many times but I cannot find an answer. On Open Site Explorer, most URLs I enter show as "No title" which means you have to hover over the URL to see which page is being referred to. I know you'd want an example, so here's your own site Moz.com 🙂 CWrkxZQ
Link Explorer | | clifra0 -
Spam Score and crawling of my site
Hello, I'm trying to analyze the spam score of my site which is 9/17 Actually I have few backlinks and all of them have a low spam score (max 4/17, just one). I think there's some kind of issue with the crawler since I get strange spam factors: Large Site with Few Links (likely true, I recently deleted a lot of tags used once) Low Number of Pages Found (wasn't it a "Large Site"??) Low Number of Internal Links (I got a considerable number) No Contact Info (I have a link to my facebook in the menu and a "contacts" page) Thin Content (It's just a blog with min 300 words per post, why thin?) Site Link Diversity is Low (likely true) Ratio of Followed to Nofollowed Subdomains (likely true) Low MozTrust or MozRank Score (true) Ratio of Followed to Nofollowed Domains (likely true) Can you please help me to understand it, is it a crawling problem or similar? If needed I will post the url of the website. Thank you so much Marco
Link Explorer | | MarcoBP0 -
Error Code 612 with robots.txt 200
Hi! I am getting this message Error Code 612: Error response for robots.txt, so the crawler do not check any page of the site. The status code for the robots.txt is 200 and it does not seem Googlebot has any problem crawling the site, so I don't know what the matter is. The site is http://www.musicopolix.com/ Thanks so much in advance for any help!
Link Explorer | | Musicopolix0 -
Why does OSE only show top 25 pages in Top Pages for Social Metrics?
Any way around this? Would like to find out the top social pages for a site. thx
Link Explorer | | IsHot0 -
Open Site Explorer not detecting linking domain on my site
So I was using Open Site Explorer to analyze my domain authority and page authority as can be seen here. My site has been covered by some press and in the article it actually links my site. Some of the articles can be seen on, here, here, and here. Any idea why none of these articles are detected as a linking domain to my site? Is there something that I am doing wrong?
Link Explorer | | herlamba1