Moz was unable to crawl your site on Jun 22, 2020\. We were unable to access your site due to a page timeout on your robots.txt, which prevented us from crawling the rest of your site.
-
Site: www.kpmg.us
Getting robots.txt timeout fail since 02/29/20. We've checked our server logs and see no errors. Went through all the steps of the "Troubleshooter".
Updated robots.txt to allow rogerbot full access:
User-agent: rogerbot
Disallow:Any ideas how to get roger to crawl my site????
-
Back to the "We were unable to access your site due to a page timeout on your robots.txt".
Could it be the sitemap.xml page specified in the robots.txt is too slow?
Sitemap: https://www.kpmg.us/sitemap.xml -
OK. Got a different error: Your site crawl timed out due to a slow server response. Passing this along to IT.
-
We fixed the situation where the robots.txt files download (see: https://www.kpmg.us/robots.txt) but rogerbot still cannot crawl the site due to some "timeout" issue on the robots.txt.
-
Hmmm, seems all our robots.txt files download as text files. But the others (ex: advisory.kpmg.us/robots.txt) work with rogerbot. I've asked our IT folk to see how were serving .txt files.
-
Hi there, thanks for reaching out!
Is the robots.txt for your site located here: "https://www.kpmg.us/robots.txt"?
If so, the issue may be that the robots.txt downloads as a text file which our crawler, rogerbot will be unable to follow. If our crawler is unable to access to the robots.txt it will cause the crawl to fail.
If you're still having issues, please feel free to reach out to the help@moz.com
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
My site links to my clients not showing in Moz
I've got a web design business with a site that's been out there for 17 years - www.biztoolsone.com. I've got links to my clients' sites in my portfolio page. The backlinks list for my clients, when I check them in Moz, don't show the links from this site. They always showed in the list until a few months ago. The site has a DA of 59 & PA (for home page) of 50. The spam score is 0/17. I'm having a tough time telling my clients that the Moz backlinks list is accurate when that site doesn't show in the list. The same is true for www.fayettevillenc.biz - this is one I created years ago to generate backlinks. It's got a spam score of 4/17, so I'm a little more understanding of that one. Thanks for any guidance!
Link Explorer | | wcksmith10 -
Is Moz's backlink checker.... just... not good?
Hey everyone! Can somebody explain to me why this keeps happening: Whenever I'm trying to backlink my competitors, I typically use RavenTools. Every time, without fail, if I put that same URL into Moz's Open Site Explorer - It gives me about 1/20th of what RavenTools shows me. Sometimes it literally comes up with 2 or 3 links total. Unfortunately, RavenTools has a cap on how many backlink checks you can perform in a month - so once I've used those up, I have to start using OSE... But, it just doesn't work. Does anyone else have this issue? Thanks!
Link Explorer | | TaylorRHawkins1 -
Inbound links not found in Moz Pro(though they exist)
These sites have many inbound links but Moz Pro finds no such links. I also used SimilarWebPro Trial to confirm the existing of such links. (these sites have IP block so it requires an VPN to access with Japan IP) osakado.cc
Link Explorer | | Yuji_m
oskd.biz Could anyone figure out why Moz can't find those links to those sites? Any tricks in place to hide those links from SeoMoz? If I pay to upgrade SimilarWebPro but Id rather stay with SeoMoz Pro. thanks in advance.0 -
Pagination & Duplicate Page Title
I was under the impression that pagination resolved duplicate page title issues with Google, but Moz is flagging them up as duplicates. Is this just a case of Moz not being able to understand pagination or am I missing something? i.e. correctly formatted pagination: http://thespacecollective.com/space-memorabilia/page/4
Link Explorer | | moon-boots1 -
Open site explorer is giving me strange redirect message.
Hi all, I tried to find answer in help section. Few people have asked this before but in none of the answers it tells how they fix it. Sorry I will need to ask here again. In Open Site Explorer i get this message. You entered the URL https://a-fotografy.co.uk/ which redirects to http://www.a-fotografy.co.uk/. Because it's likely to have more accurate metrics, we're showing data for the redirected URL instead. Click here to analyze https://a-fotografy.co.uk/ instead? My current site is https://a-fotografy.co.uk/ not sure why it is redirecting to my old domain. Weird. My web developer couldn't explain this issue. Saying that I also go my first analytics report and in priority issues I have http://a-fotografy.co.uk 302 redirect and not sure where it is redirecting to. If someone could shed some light of how to figure out how to find exactly what is redirecting to what and where is the problem. Thank you guys, Regards, Armands
Link Explorer | | A_Fotografy0 -
How is Moz Page Authority Calculated?
Can anyone refer me to a document that explains the measures that go into the calculation of Moz Page Authority? I'm chiefly interested in what counts and what doesn't -- follow links, no-follow links, internal and external, what (if any) on-site factors, etc.
Link Explorer | | GlennFerrell0 -
My linked in profile is showing up the opensite exploer but not showing up in on page grader, any ideas why? the onpage grader says in url is in accessible
my linked on company profile ios not showing up in the on page grader tool, it return un accessible uRL? but we give me metrics in domain checker? its a new profile i set up about 8 hours ago,
Link Explorer | | maxmxmax0 -
Removing the clutter of site-wide links
I have a multi-part question with regard to the moz link index and some presentation suggestions. Firstly I would be interested to know how the link index treats site-wide links with regard to metrics such as DA, and PA. We all know that it is highly likely that SE's are unlikely to pass full link value across from sitewide links, and therefore it would make sense for Moz values to account for this as well - if they do not already. One annoying thing that also relates to sitewides is that they tend to clutter the much of the information presentation in a few of the tools (you can't see wood for trees as it were). This is most prominent in the "Just Discovered" page - if you have a sitewides on a large site, you can often find that this screen is just totally filled with these links as they are found. It would be very useful to be able to filter these out, as they are of little interest - currently I can't see a way of filtering them out. A further value where they create to much noise is the 'Total Links' value. Where sitewides are included in this value, the value actually becomes pretty meaningless as you can find that the majority of that value is sitewides. It would therefore be useful if there was another value for 'Total Links - Excluding Sitewides' where maybe value of 1 was just added to the count for a site wide
Link Explorer | | James770