Moz was unable to crawl your site on Jun 22, 2020\. We were unable to access your site due to a page timeout on your robots.txt, which prevented us from crawling the rest of your site.
-
Site: www.kpmg.us
Getting robots.txt timeout fail since 02/29/20. We've checked our server logs and see no errors. Went through all the steps of the "Troubleshooter".
Updated robots.txt to allow rogerbot full access:
User-agent: rogerbot
Disallow:Any ideas how to get roger to crawl my site????
-
Back to the "We were unable to access your site due to a page timeout on your robots.txt".
Could it be the sitemap.xml page specified in the robots.txt is too slow?
Sitemap: https://www.kpmg.us/sitemap.xml -
OK. Got a different error: Your site crawl timed out due to a slow server response. Passing this along to IT.
-
We fixed the situation where the robots.txt files download (see: https://www.kpmg.us/robots.txt) but rogerbot still cannot crawl the site due to some "timeout" issue on the robots.txt.
-
Hmmm, seems all our robots.txt files download as text files. But the others (ex: advisory.kpmg.us/robots.txt) work with rogerbot. I've asked our IT folk to see how were serving .txt files.
-
Hi there, thanks for reaching out!
Is the robots.txt for your site located here: "https://www.kpmg.us/robots.txt"?
If so, the issue may be that the robots.txt downloads as a text file which our crawler, rogerbot will be unable to follow. If our crawler is unable to access to the robots.txt it will cause the crawl to fail.
If you're still having issues, please feel free to reach out to the help@moz.com
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why is my backlink not detected here, but it appears on google search console? I have added the moz robots
Why is my backlink not detected here, but it appears on google search console? I have added the moz robots
Link Explorer | | gunawanjord0 -
The New Link Explorer (which will replace Open Site Explorer) is Now in Beta
Howdy Moz friends, Today, the Moz team is making a new tool -- Link Explorer -- available in private beta for Moz Pro subscribers (including those taking a free trial). The tool is still a little ways out from public launch, but we wanted to get your feedback to help make it the best product it can be. What's Link Explorer? In essence, it's a replacement for Open Site Explorer (Moz's tool for link discovery, competitive analysis, and link building) that addresses many of its most pernicious challenges, such as: Daily updates - no longer will you need to wait a month or two to see a new DA score or the links you built last week. Link Explorer updates every 24 hours with all the new links we've discovered that day, and gives a new DA score each night. A MUCH bigger index - OSE has always been known for having quality links, but quantity has been an issue. No longer. The new tool's link index is more than 20 times larger than Open Site Explorer's, covering trillions of links across hundreds of billions of pages, while maintaining a focus on high quality domains and pages. Additional functionality - new graphs (like link growth over time), new charts (like gained+lost links), new filters and sorts, and some new kinds of data coming soon. Improved metrics - Domain Authority and Page Authority have both been upgraded to have better correlations with Google's rankings (and they now update every 24 hours) Insanely fast - page load times on the new tool are almost as fast as Google's results 🙂 Less time waiting means more time to dig into results Link Tracking Lists - check a box next to any links of interest and you can build lists in the tool to track them over time, see whether/when they link, prioritize your outreach efforts, and (in the future) get aggregated data and alerts about those links There's much more to come, but we'd love for you to check out Link Explorer, find bugs, report things you love (and don't), and help us make it the best possible product for you and your teams. You can leave feedback here in this Q+A thread, email help@moz.com, or send feedback through the feedback form in Link Explorer.
Link Explorer | | randfish18 -
403 errors in Moz but not in Google Search Console
Hello, Moz is showing that one of the sites I manage has about ten 403 errors on main pages, including the home page. But when I go to Google Search Console, I'm not getting any 403 errors. I don't know too much about this site (I handle the SEO for a few sites as a contractor for a digital marketing agency), but I can see that it's a WordPress site (I'm not sure if that's relevant). Can I assume this a Moz issue only? Thanks, Susannah Noel
Link Explorer | | SusannahK.Noel0 -
MOZ Rank Or MOZ Wrong!?
Hi everyone, I would like to show up you a majestic mistake of ranking. I've checked some of my competitors and encountered that site has no DmT(Domain Trust Rank) but its mT(Moz Trust) is 3.83.Its good! but its not good! which one is true? is it possible that a domain isn't trust but a home page is trust!? zv99ZJV zv99ZJV
Link Explorer | | abolfazli0 -
Duplicated content detected with MOZ crawl with canonical applied
Hi there! I have a slight problem.
Link Explorer | | Eurasmus.com
I have a site with Joomla 3.3 that we recently migrated from 2.5. Joomla, for some reason that I don´t really get, creates hundreds of weird urls for the site like
mydomain.com/en -> joomla creates en/home/149-xxx-xxx/xxxxxx-xxxxxx that links to the first one.
The new version 3.3 knows this bug and applies a rel=canonical to the ones created "artificially", so they should not be identified as duplicated. Sample piece of code: en/home/149-all-en/xxxxxxx-xxxxxx" rel="canonical" / MOZ crawler identifies this as duplicated and like this I have thousands of pages duplicated all with titles, content etc... all the ones created by joomla. Still my site has good SEO results and I can not see any penalties but I am a bit concerned they may come in the future.... Can anyone explain me what is happening? Thank you in advance for your time,0 -
Open Site Explorer Question
Hello Mozers. It has been a while 🙂 OK - I run an open site explorer report, toggle over to 'Linking Domains' because I am more interested in the number of actual websites that link to my client rather than the number of links my client has (and of course what websites are linking) and although I have been doing this for years there is one little thing I am a bit confused about. The report shows (from left to right) the 'Linking Root Domain", the "Domain Authority" for the linking domain and then the 'Number of Linking Root Domains" for each 'Linking Root Domain" My question is (rather stupid, I am sure) is the number of linking root domains for each linking root domain the number of websites linking to them OR the number of websites to which they link?
Link Explorer | | Vizergy1 -
Open Site Explorer
I have entered my site (patioenjoyment.com) a few different times into the OSE and it never seems to find it, MOZ has found it because it has done the crawl test for me a few times, found some issues and I have been working on those. But I assume that if it is able crawl my site it should find it to run the OSE. It has come back with "**It looks like we haven't discovered established link data for this URL yet." everytime. ** Thanks, Julie
Link Explorer | | patioenjoyment0 -
Does the inbound links report include links to all pages of the domain being researched?
If I enter 'abc.com' am I only getting results for 'abc.com' or will I get results for the internal pages of 'abc.com' as well (i.e. 'abc.com/page1.html)? There is a bit of a discrepancy between these results and inbound link results in semrush for example. Then again it seems whenever you use different tools to measure the same thing you get wildly varying results. How do you all deal with that?
Link Explorer | | AISEO0