Moz unable to crawl my Zenfolio website
-
Hey guys,
I am attempting to optimize a website for my wife's business but Moz is unable to crawl it. Zenfolio is the web hosting service (she is a photographer).
The error message is:
**Moz was unable to crawl your site on Apr 1, 2019. **Our crawler was not able to access the robots.txt file on your site. This often occurs because of a server error from the robots.txt. Although this may have been caused by a temporary outage, we recommend making sure your robots.txt file is accessible and that your network and server are working correctly. Typically errors like this should be investigated and fixed by the site webmaster. Read our troubleshooting guide.
I did read the troubleshooting guide but nothing worked.
My robots.txt file disallows a few bots, but not roger bot. Anyone have any idea what is going on? Or do I need to request server logs from Zenfolio?
Thanks
-
It appears that Zenfolio really doesn't want Moz crawling their site for some reason. They want me to jump through hoops to get it to work and, being an ultra small business, cant justify the effort to get it working.
Guess I could always migrate to another web host.
Thanks for the replies.
-
Hi OP.
Hopefully you got in touch with Dave and the Moz team, and you have got your answer!
I haven't got a direct answer but from experience of Moz and other crawler softwares, my money would be on a temporary issue with your server being unresponsive.
As per the "Moz Can't Crawl your site guide, as it states, if it can't read your robots.txt file, then "Roger" won't crawl your site.
This is my simplest explanation I have
Hopefully it's resolved soon!
P.S
I looked at Zenfolio, if it's a self-hosted CMS system, it might be worth checking they haven't got any special rules in place to block Moz and other popular crawlers
-
Hey! Dave here from the Help Team
Can you write into us at help@moz.com with the site that is having crawl issues? We can take a look and help you out.
thanks!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
MOZ point
I don't know why, but since a week ago I'm not receiving moz point for my activity on moz forum. Example
Getting Started | | Roman-Delcarmen
Today I posted 3 answer in the Question section but in my moz profile does not show the 3 moz point that normally I receive for that. I week ago suddenly I received 20 moz point, why I dont have any idea, maybe someone mark one of my asnwer as good answer. So my point is where I cant found the exact tracking record of my activity1 -
Can't track my site, keep getting "Ooops. Our crawlers are unable to access that URL"
Hello, So i keep getting this message and I went to hurl.it and I get 200 response. But it appears its not my actual homepage bc it says the body is empty and in the title it says "COMING SOON" which is not what my actual homepage says. Does anyone know what this means?? Thank you in advance! Rena
Getting Started | | Palila-Studio0 -
Our website has 8 subdomains for each country, do i need to set up a campaign for each? And have to upgrade to have more than 5 campaigns?
Hello there, Our website is set up to have one main domain with 8 subdomains for each country. To be able to track each subdomain do i need 8 campaigns when setting up in moz?
Getting Started | | cwpang0 -
New to Moz Pro? Join our upcoming free webinar this Friday!
Hello everyone! We'll be holding a webinar on Friday to help new members learn about what all Pro has to offer, show some off our most popular tools, and get you comfortable with the dashboard. Register here**: https://www3.gotomeeting.com/register/132016958** Date: Friday, October 17th Time: 10:00 AM - 11:00 AM PDT Hope to see you all there! If you can't make it, you can watch a previous webinar here: http://moz.com/help/guides/getting-started
Getting Started | | jennita4 -
Moz Staff: History + Watch List
Hi All, I have been going crazy trying to find a thread I read last week ... I think!
Getting Started | | Mark_Ch
Several search attempts later, no closer to finding the thread in question. A genius moment or simple frustration got me thinking. How cool would it be to have the following features on MOZ Q&A Forum. Watch List
1] Check off any threads that you would like to monitor over time. Should you stumble upon a thread of value then simply add to watch list and view it's progress over time. No searching for the thread in question. History
2] Keep a history off all threads viewed over a period of time. Thanks Mark0 -
Why can't I add more than one campaign in my trial version of MOZ?
Hi, I'm on the 30 day trial and I can only run one campaign right now. I thought I could add in 5 campaigns. When I go into Manage Campaigns the 'add a campaign' button is light blue and I can't click it. Is this just because I'm on the trial? Or should I be able to add 4 more campaigns?
Getting Started | | Sophie-Kool0 -
Moz Analytics - Custom Report Question
Hello Moz! I haven't been here since the days of SEOMoz, but it's great to be back and i'm really pleased to see lot of shiny new tools at my disposal, but I do have a question in regards to functionality. If I have, for example, 15 clients in my dashboard and would like to run the same report for all of them. Assuming that I just finished creating a custom report for my first client, how can I duplicate this report without having to re-create this report 15 times? Additionally, while I am here. I have noticed that when I add a campaign, the last screen never finishes loading and the window seems to freeze. A quick refresh fixes this, but I thought i'd flag it 🙂
Getting Started | | Paul_Tovey0 -
Can I use wildcards "*" when setting up a new Moz campaign?
Basically I would like the Moz crawler to focus on a specific section of our domain. We do not bucket things via folder groups, so the use of wildcards would be applicable to us. Our URL structure: www.domain.com/some-stuff-here/p12345 Is the example below a valid input to track the above URL structure? www.domain.com//p Thanks.
Getting Started | | WEB-IRS0