Moz was unable to crawl your site on Jun 22, 2020\. We were unable to access your site due to a page timeout on your robots.txt, which prevented us from crawling the rest of your site.
-
Site: www.kpmg.us
Getting robots.txt timeout fail since 02/29/20. We've checked our server logs and see no errors. Went through all the steps of the "Troubleshooter".
Updated robots.txt to allow rogerbot full access:
User-agent: rogerbot
Disallow:Any ideas how to get roger to crawl my site????
-
Back to the "We were unable to access your site due to a page timeout on your robots.txt".
Could it be the sitemap.xml page specified in the robots.txt is too slow?
Sitemap: https://www.kpmg.us/sitemap.xml -
OK. Got a different error: Your site crawl timed out due to a slow server response. Passing this along to IT.
-
We fixed the situation where the robots.txt files download (see: https://www.kpmg.us/robots.txt) but rogerbot still cannot crawl the site due to some "timeout" issue on the robots.txt.
-
Hmmm, seems all our robots.txt files download as text files. But the others (ex: advisory.kpmg.us/robots.txt) work with rogerbot. I've asked our IT folk to see how were serving .txt files.
-
Hi there, thanks for reaching out!
Is the robots.txt for your site located here: "https://www.kpmg.us/robots.txt"?
If so, the issue may be that the robots.txt downloads as a text file which our crawler, rogerbot will be unable to follow. If our crawler is unable to access to the robots.txt it will cause the crawl to fail.
If you're still having issues, please feel free to reach out to the help@moz.com
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
My Site Facing indexing issues
Hello, I am facing indexing issues on one of my which is about budget bushcraft knife , Four months have been passed I built my site and published almost 7 article so far. But i am worried no any single keyword ranked on google till yet as i checked through MOZ site explorer. Can anyone guide me what should I need to do now. Thanks
Link Explorer | | james112230 -
Crawling 4XX errors because of URL accented
Hello guys, I am experiencing crawling errors in Moz because the URLs read by spiders contain accents and special characters, which I know isn't best but yet my client needs to keep it. I know that Moz "uses percent encoding to parse the HTML in the source code, so any line breaks and spaces in your HTML links or sitemap links are converted to %0A and %20, causing a 404 error". Is there any way to avoid these errors happening in the dashboard? Or am I supposed to simply ignore it?
Link Explorer | | fran8751 -
How MOZ OSE Work and calculate Compare Links Metrics ?
I want to know everything about compare links metrics. Like how Moz work and calculate this value of Internal Equity-Passing Links External Equity-Passing Links Total Equity-Passing Links Total Internal Links Total External Links Total Links in each metrics ( Page Specific Metrics: Subdomain Metrics: Root Domain Metrics: ) Each metrics Show different value for same metrics.
Link Explorer | | HuptechWebseo0 -
Learn how to use Open Site Explorer's Just-Discovered Links report to build links and manage your brand. Get your Daily SEO Fix!
The Just-Discovered Links report in Open Site Explorer helps you discover recently created links--within an hour of them being published. It's pretty nifty. In today's Daily SEO Fix, Nick shows you how to use the report to view who is linking to you, how they're doing it, and what they are saying so you can capitalize on link opportunities while they're still fresh plus, monitor and participate in the conversations happening about your brand. Watch our tutorial Using Open Site Explorer to Find Fresh Links and Manage Your Brand Online and get ready to dive in! This video is part of The Moz Daily SEO Fix tutorial series--Moz tool tips and tricks in under 2 minutes. To watch all of our videos so far, and to subscribe to future ones, make sure to visit the Daily SEO Fix channel on YouTube.
Link Explorer | | kellyjcoop3 -
Use Open Site Explorer and the Keyword Difficulty Tool to find your competitors' keywords and how they're ranking for them. Get your Daily SEO Fix!
In today's Daily SEO Fix, Jacki walks through using Open Site Explorer's anchor text report to find keywords your competitors may be targeting, and how to use the Keyword Difficulty Tool to tease out what's helping them rank. Watch "Keyword Research with OSE and the Keyword Difficulty Tool" now! The Daily SEO Fix is an ongoing series of Moz tool tips and tricks in under 2 minutes. To watch all of our videos so far, and to subscribe to future ones, make sure to visit the Daily SEO Fix channel on YouTube. If you'd like a more in-depth guide to using the Keyword Difficulty Tool and its Full SERP Analysis Report for competitive insights, check out Cyrus Shepard's excellent Moz Academy video on the subject.
Link Explorer | | MattRoney1 -
Page Authority Migration
PageA.html has a lot of links and has a Page Authority score. It is 301 redirected to PageB.html and eventually in OSE this also accumulates a Page Authority score. Is there a reason why PageA.html many months/years after continues to have a Page Authority when it is expected that its Authority signals have migrated (with any dampening effect etc) across to PageB.html.
Link Explorer | | Infotex_UK0 -
Why won't Open Site Explorer detect inbound links from big name websites?
I understand smaller, spammy sites woudn't get indexed, but some of the websites linking to mine have high domain authorities. The website is a parked subdomain.
Link Explorer | | myers3680 -
Open Site Explorer Only showing 2 Internal Links
Hi, for some reason Open Site Explorer only shows 2 internal links linking to our homepage (and other pages). When looking at a competitors site, they have thousands of internal links to their homepage! Are there any reasons why OSE is only showing 2 for our site, when in fact there are hundreds and thousands? Thanks.
Link Explorer | | Jaybeamer0