Moz was unable to crawl your site on Jun 22, 2020\. We were unable to access your site due to a page timeout on your robots.txt, which prevented us from crawling the rest of your site.
-
Site: www.kpmg.us
Getting robots.txt timeout fail since 02/29/20. We've checked our server logs and see no errors. Went through all the steps of the "Troubleshooter".
Updated robots.txt to allow rogerbot full access:
User-agent: rogerbot
Disallow:Any ideas how to get roger to crawl my site????
-
Back to the "We were unable to access your site due to a page timeout on your robots.txt".
Could it be the sitemap.xml page specified in the robots.txt is too slow?
Sitemap: https://www.kpmg.us/sitemap.xml -
OK. Got a different error: Your site crawl timed out due to a slow server response. Passing this along to IT.
-
We fixed the situation where the robots.txt files download (see: https://www.kpmg.us/robots.txt) but rogerbot still cannot crawl the site due to some "timeout" issue on the robots.txt.
-
Hmmm, seems all our robots.txt files download as text files. But the others (ex: advisory.kpmg.us/robots.txt) work with rogerbot. I've asked our IT folk to see how were serving .txt files.
-
Hi there, thanks for reaching out!
Is the robots.txt for your site located here: "https://www.kpmg.us/robots.txt"?
If so, the issue may be that the robots.txt downloads as a text file which our crawler, rogerbot will be unable to follow. If our crawler is unable to access to the robots.txt it will cause the crawl to fail.
If you're still having issues, please feel free to reach out to the help@moz.com
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz's new Link Explorer displaying the DA marginally less than Site Explorer
Moz's new Link Explorer displaying the DA marginally less than Site Explorer. Old one is showing it 46 while new link explorer is showing the DA as 40.
Link Explorer | | dhananjay.kumar11 -
After how long Moz show the matrices of a new domain
I want to know the Matrices of this site " Clash of lights ". Its a new domain how can i find its matrices like DA PA etc in Moz. Please guide me through it.
Link Explorer | | Muhammadahamd0 -
How to index backlinks in moz?
hello my website has some backlinks from high authority website that has been indexed by google but moz is not indexing my backlinks and also my pages so what should i do?
Link Explorer | | yuewiioweiwjksa38320 -
How Do You Deal With Duplicate Content On A Retail Site
Hi Guys We make custom portfolios and boxes and every time we have a Moz site crawl, Rogerbot always returns a number of Duplicate Content Issues relating to different products. As per the image below (which I hope is visible?!) Rogerbot has flagged duplicate content onto one product that relates to 5 other different products. For instance there is duplicate content for an A4 Leather Portfolio and an A3 Leather Portfolio and an 11"x17" Leather Portfolio. I can't redirect or canonicalise to just the A4 Portfolio as they are all individually different products. The information on each page although similar, is relevant to each of the products, so rewriting a different blurb on each product page, will not be user friendly for our customers. I could ignore the duplicate content issues, but then that isn't good practise (and also makes for a very unsatisfactory looking Dashboard!) Any ideas?? Nick wzQLgBl
Link Explorer | | nick_HandCo1 -
Open site explorer is giving me strange redirect message.
Hi all, I tried to find answer in help section. Few people have asked this before but in none of the answers it tells how they fix it. Sorry I will need to ask here again. In Open Site Explorer i get this message. You entered the URL https://a-fotografy.co.uk/ which redirects to http://www.a-fotografy.co.uk/. Because it's likely to have more accurate metrics, we're showing data for the redirected URL instead. Click here to analyze https://a-fotografy.co.uk/ instead? My current site is https://a-fotografy.co.uk/ not sure why it is redirecting to my old domain. Weird. My web developer couldn't explain this issue. Saying that I also go my first analytics report and in priority issues I have http://a-fotografy.co.uk 302 redirect and not sure where it is redirecting to. If someone could shed some light of how to figure out how to find exactly what is redirecting to what and where is the problem. Thank you guys, Regards, Armands
Link Explorer | | A_Fotografy0 -
Why is Open Site Explorer Showing me Spammy Links That I Already Removed?
Every two months or so, I like to download the OSE list of all the backlinks passing equity to my site. I do this so I can see which spammy links I need to try and remove. However, I see that OSE is still showing me links that I already removed...Either A) the page that is being linked to is 404-ed B) the spammy website that was linking doesn't exist anymore or C) the spammy site still exists, but the link is GONE. Why is OSE still spidering these sites and links? Does that mean Google is still spidering these links? Should I disavow them to be safe? Thank you! P.S We are not experiencing any drastic drops in traffic or penalties at the moment.
Link Explorer | | TMI.com0 -
Duplicate Home Page
'Content that is identical (or nearly identical) to content on other pages of your site forces your pages to unnecessarily compete with each other for rankings.' Can anyone tell me why this is the case? I know for a fact that their are no other duplicates of my home page, however moz disagrees. Cheers for any help. **Taylor **
Link Explorer | | Taylor1230
