Moz unable to crawl my Zenfolio website
-
Hey guys,
I am attempting to optimize a website for my wife's business but Moz is unable to crawl it. Zenfolio is the web hosting service (she is a photographer).
The error message is:
**Moz was unable to crawl your site on Apr 1, 2019. **Our crawler was not able to access the robots.txt file on your site. This often occurs because of a server error from the robots.txt. Although this may have been caused by a temporary outage, we recommend making sure your robots.txt file is accessible and that your network and server are working correctly. Typically errors like this should be investigated and fixed by the site webmaster. Read our troubleshooting guide.
I did read the troubleshooting guide but nothing worked.
My robots.txt file disallows a few bots, but not roger bot. Anyone have any idea what is going on? Or do I need to request server logs from Zenfolio?
Thanks
-
It appears that Zenfolio really doesn't want Moz crawling their site for some reason. They want me to jump through hoops to get it to work and, being an ultra small business, cant justify the effort to get it working.
Guess I could always migrate to another web host.
Thanks for the replies.
-
Hi OP.
Hopefully you got in touch with Dave and the Moz team, and you have got your answer!
I haven't got a direct answer but from experience of Moz and other crawler softwares, my money would be on a temporary issue with your server being unresponsive.
As per the "Moz Can't Crawl your site guide, as it states, if it can't read your robots.txt file, then "Roger" won't crawl your site.
This is my simplest explanation I have
Hopefully it's resolved soon!
P.S
I looked at Zenfolio, if it's a self-hosted CMS system, it might be worth checking they haven't got any special rules in place to block Moz and other popular crawlers
-
Hey! Dave here from the Help Team
Can you write into us at help@moz.com with the site that is having crawl issues? We can take a look and help you out.
thanks!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Is there anyway to cancel moz local?
Is there anyway to cancel moz local? I can not figure it out. I've tried for hours now.
Getting Started | | SouthBayWebs0 -
Moz not able to crawl our site - any advice?
When I try and crawl our site through Moz it gives this message: Moz was unable to crawl your site on Aug 7, 2019. Our crawler was banned by a page on your site, either through your robots.txt, the X-Robots-Tag HTTP header, or the meta robots tag. Update these tags to allow your page and the rest of your site to be crawled. If this error is found on any page on your site, it prevents our crawler (and some search engines) from crawling the rest of your site. Typically errors like this should be investigated and fixed by the site webmaster. I have been through all the help and doesn't seem to be any issues. You can check the site and robots.txt here: https://myfamilyclub.co.uk/robots.txt. Anyone got any advice on where I could go to get this sorted?
Getting Started | | MyFamilClubLtd1 -
Does the Moz crawler have a static ip address?
We block AWS from crawling our site. Does the moz crawler use a static ip address that we could allow? I'm currently not able to add a campaign because Moz can't connect to our site.
Getting Started | | uShip0 -
MOZ Removes WWW
I just signed up for Moz... Created my campaign. I'm noticing Moz is removing the www* from my domain. When they do analysis its wrong. when i go to google and type site:www.domain.com i have 350,000 + pages. MOZ is telling me i have 1,277 pages because its getting the external links by using *domain.com and not www.domain.com how do i force MOZ to read my domain as www.domain.com and not *.domain.com
Getting Started | | CarlosJaa0 -
New to MOZ and working with Web Mentions. Can I use operators?
Our name is HostDime but often put as Host Dime (2 words) by news sources and other sites. How do I set up my brand mention so I only get a notice when both words appear, in order, together. I don't want "That host is a dime" and such. Can I use a +Host +Dime?"Host Dime"? Do these operators work in MOZ?
Getting Started | | hostdime0 -
Moz Analytics set up: should I use www or not when I do website set up?
Hello, When doing website set up will www.mysite.com yield different results than just mysite.com ? Same about competitors: should I use www or not when I set up their sites? Or it won't matter? Thanks,
Getting Started | | dstepans
Denis0 -
Campaign.crawl-seed.bad-response ???
Hello Guys I have just tried to set-up a new campaign for this site http://www.emsababies.co.uk/ HoweverI keep getting this error ?? campaign.crawl-seed.bad-response Anyone know what I am doing wrong ?? Cheers James
Getting Started | | BlueNinja1
