Moz was unable to crawl your site on Jun 22, 2020\. We were unable to access your site due to a page timeout on your robots.txt, which prevented us from crawling the rest of your site.
-
Site: www.kpmg.us
Getting robots.txt timeout fail since 02/29/20. We've checked our server logs and see no errors. Went through all the steps of the "Troubleshooter".
Updated robots.txt to allow rogerbot full access:
User-agent: rogerbot
Disallow:Any ideas how to get roger to crawl my site????
-
Back to the "We were unable to access your site due to a page timeout on your robots.txt".
Could it be the sitemap.xml page specified in the robots.txt is too slow?
Sitemap: https://www.kpmg.us/sitemap.xml
-
OK. Got a different error: Your site crawl timed out due to a slow server response. Passing this along to IT.
-
We fixed the situation where the robots.txt files download (see: https://www.kpmg.us/robots.txt) but rogerbot still cannot crawl the site due to some "timeout" issue on the robots.txt.
-
Hmmm, seems all our robots.txt files download as text files. But the others (ex: advisory.kpmg.us/robots.txt) work with rogerbot. I've asked our IT folk to see how were serving .txt files.
-
Hi there, thanks for reaching out!
Is the robots.txt for your site located here: "https://www.kpmg.us/robots.txt"?
If so, the issue may be that the robots.txt downloads as a text file which our crawler, rogerbot will be unable to follow. If our crawler is unable to access to the robots.txt it will cause the crawl to fail.
If you're still having issues, please feel free to reach out to the help@moz.com
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why is my backlink not detected here, but it appears on google search console? I have added the moz robots
Why is my backlink not detected here, but it appears on google search console? I have added the moz robots
Link Explorer | | gunawanjord0 -
Why Moz is not providing keywords data above 300K
When we extract keywords of a website, moz shows us 300K results, they are enough but, why moz is not showing all keywords data
Link Explorer | | nkodiou1 -
Is there any study or detailed report on How Moz & Google calculates domain authority ?
I want to know, Is there any detailed report or study done by someone on the Domain authority calculations between Google & Moz. Mainly, I would like to know 1. How Google calculates Domain Authority and How Moz does ? 2. Is there any correlation in majority of the cases or both differs vastly ?
Link Explorer | | sssnee20 -
Angular SPA & MOZ Crawl Issues
Website: https://www.exambazaar.com/ Issue: Domain Authority & Page Authority 1/100 I am using Prerender to cache/render static pages to crawl agents but MOZ is not able to crawl through my website (https://www.exambazaar.com/). Hence I think it has a domain authority of 1/100. I have been in touch with Prerender support to find a fix for the same and have also added dotbot to the list of crawler agents in addition to Prerender default list which includes rogerbot. Do you have any suggestions to fix this? List: https://github.com/prerender/prerender-node/commit/5e9044e3f5c7a3bad536d86d26666c0d868bdfff Adding dotbot to Express Server:
Link Explorer | | gparashar
prerender.crawlerUserAgents.push('dotbot');0 -
Spam Score and crawling of my site
Hello, I'm trying to analyze the spam score of my site which is 9/17 Actually I have few backlinks and all of them have a low spam score (max 4/17, just one). I think there's some kind of issue with the crawler since I get strange spam factors: Large Site with Few Links (likely true, I recently deleted a lot of tags used once) Low Number of Pages Found (wasn't it a "Large Site"??) Low Number of Internal Links (I got a considerable number) No Contact Info (I have a link to my facebook in the menu and a "contacts" page) Thin Content (It's just a blog with min 300 words per post, why thin?) Site Link Diversity is Low (likely true) Ratio of Followed to Nofollowed Subdomains (likely true) Low MozTrust or MozRank Score (true) Ratio of Followed to Nofollowed Domains (likely true) Can you please help me to understand it, is it a crawling problem or similar? If needed I will post the url of the website. Thank you so much Marco
Link Explorer | | MarcoBP0 -
SEO report for specific pages (not just based on keyword analysis)
Hi, i am after a report where i can type in a URL nd get SPECIFIC details on how to exactly optimise a SPECIFIC page (not just the home page) for better organic results. A complete overview to everything in relation to tags, images tags, headers etc etc. I am yet to find the ability to get such a report with your service. COudl you please advise how i can do so. Thanks
Link Explorer | | andrewlos0 -
Is there a preferred Browser for Moz?
I having difficulties using Firefox on the Mac. Crawl Diagnostics results never show up. I see the numbers of errors but can't see what the errors are. Appreciate any help. I'm a newbie.
Link Explorer | | pompanoartist0 -
Unable to download CSV Report
Hi, I have the email notifying me that my report is ready to download but the link simply opens a blank webpage. If I go to 'Recent CSV Reports' I can see my report listed but if I click the download button nothing happens. Can anyone suggest a solution? Thanks Hugh
Link Explorer | | HLConsulting0