Moz unable to crawl my Zenfolio website
-
Hey guys,
I am attempting to optimize a website for my wife's business but Moz is unable to crawl it. Zenfolio is the web hosting service (she is a photographer).
The error message is:
**Moz was unable to crawl your site on Apr 1, 2019. **Our crawler was not able to access the robots.txt file on your site. This often occurs because of a server error from the robots.txt. Although this may have been caused by a temporary outage, we recommend making sure your robots.txt file is accessible and that your network and server are working correctly. Typically errors like this should be investigated and fixed by the site webmaster. Read our troubleshooting guide.
I did read the troubleshooting guide but nothing worked.
My robots.txt file disallows a few bots, but not roger bot. Anyone have any idea what is going on? Or do I need to request server logs from Zenfolio?
Thanks
-
It appears that Zenfolio really doesn't want Moz crawling their site for some reason. They want me to jump through hoops to get it to work and, being an ultra small business, cant justify the effort to get it working.
Guess I could always migrate to another web host.
Thanks for the replies.
-
Hi OP.
Hopefully you got in touch with Dave and the Moz team, and you have got your answer!
I haven't got a direct answer but from experience of Moz and other crawler softwares, my money would be on a temporary issue with your server being unresponsive.
As per the "Moz Can't Crawl your site guide, as it states, if it can't read your robots.txt file, then "Roger" won't crawl your site.
This is my simplest explanation I have
Hopefully it's resolved soon!
P.S
I looked at Zenfolio, if it's a self-hosted CMS system, it might be worth checking they haven't got any special rules in place to block Moz and other popular crawlers
-
Hey! Dave here from the Help Team
Can you write into us at help@moz.com with the site that is having crawl issues? We can take a look and help you out.
thanks!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Can Moz Monitor a JS Site?
We are building a new site that, on the blog landing page, uses JS to populate the individual blog article links on the page. The links are not viewable in the page source, but do appear once the page fully renders. After running an on-demand crawl of the site (in QA...not indexed yet), it appears that Moz isn't indexing these pages, and it also isn't reading other page elements that load later (like an H1 that is rendered in JS but not in page source). Are we going to be able to use Moz to track this site? Is there some setting to help?
Getting Started | | Rodrigo-DC0 -
Clarifications on the Moz Analytics package (Medium - $149 per month)
What are the Moz tools available with this package? What factors of SEO can be checked with these tools? With this package, is it possible to provide a single URL (preferably home page) and Moz will analyse the entire site and highlight how the site performs wrt various SEO factors? This package states that with this package we can run 10 Moz Analytics campaigns. Our understanding of Moz Analytics Campaign is every site; say www.test.com is one analytics campaign. Are we correct? Does the subdomains within a parent domain also considered as one analytics campaign. For e.g., if I have sites: www.mydomain.com and www.xxx.mydomain.com are they considered two separate campaigns or are they considered as one single campaign? In this package it is listed as 750 keywords, what does this signify? In what way this feature can be used to check our site’s SEO compliance. Please elaborate. In this package it is listed as 15 social accounts, what does this signify? In what way this feature can be used to check our site’s SEO compliance. Please elaborate. What do you mean by branded reports?
Getting Started | | WebCCTrial0 -
Why can't I add more than one campaign in my trial version of MOZ?
Hi, I'm on the 30 day trial and I can only run one campaign right now. I thought I could add in 5 campaigns. When I go into Manage Campaigns the 'add a campaign' button is light blue and I can't click it. Is this just because I'm on the trial? Or should I be able to add 4 more campaigns?
Getting Started | | Sophie-Kool0 -
Downloading Moz.com data?
Is it possible to download the data in my Moz analytics for use in Excel? I'm specifically after the data behind the graphs in the Links section. I don't see a way to download this...
Getting Started | | IrvCo_Interactive0 -
How to get moz to crawl a staging domain that is blocked by robots.txt
Is it possible to get Moz to do a crawl report on a domain blocked by robots.txt and actually display all errors instead of only one saying the domain was blocket in robots.txt? Anything i can add to robots.txt to make moz able to do the crawl report but still hinder google from crawling a staging domain?
Getting Started | | classifiedtech0 -
Why does moz show "not in top 50" for all my keywords???
Hello, I signed up to moz pro 4 days ago. And so far it seems to be tracking visits etc. But all my keywords say "not in top 50" . Why is this? Is this normal? Just to confirm most of the keywords i pasted in from my webmaster tools and i only chose the ones that were in top 50
Getting Started | | casper09030 -
Can MOZ crawl our website twice in a week?
I want to generate MOZ crawl errors report twice in a week. Is it possible to do that.
Getting Started | | chandman0 -
Whenever I try to access campaigns in moz pro I get an error page
I recently signed-up for a new pro account. As I was adding my first subdomain everything was fine until I was asked to link to GA, when I clicked yes I got this error message: 403 Forbidden Now every time I click on set-up campaign I get taken to a page with nothing but the 403 Forbidden text.
Getting Started | | Toptal0