Welcome to the Q&A Forum

Hey Jansen,

A few tips from me, hope it helps:
1. You don't need to pause CloudFlare completely, you can go into DNS and change the orange "cloud" to a grey one which will disable CF only on that record
2. Initially I thought about creating a new record for Moz which I could use for indexing (i.e. moz.yourdomain.com) but that will lose all search ranking records so not a good idea
3. Moz definitely works with CF + SSL, I have Moz it working on theright.fit domain, I reckon with a bit more help and testing we can get to the bottom of it, if you got time let's put together a complete list of settings and see where mine differs from yours
4. The domain has Mirrage ON, RocketLoader OFF and Minify ON (all options)

Hey Jorge,

Sorry to hear that mate. I remember my first tests failed too, how long has it been since you upgraded to CloudFlare Pro (you said Plus but I cannot find this plan on their plan list)?
Moz has a DNS cache on their side to speed up the requests which took a week to refresh on mine. Maybe just wait a couple of more days and give it another go.
I remember giving up and then one day, out of the blue, it started working and it's been like that since.

Regarding the theme, glad you liked it! But yes, it's fully customized. We started with a bunch of plugins but now we pretty much just have our custom plugin which is deeply integrated with the theme, also customized. We based it off Underscores but I don't think there is much code left from it either.

Is your site empleostampico.com? The css isn't loading for me in AU, neither the js or the images.
Wondering if it's something to do with that.

Sorry Jorge, not sure why I called you Rodrigo...
No worries, happy to help, it drove me crazy too

So, I reviewed my CloudFlare settings, I got firewall security level set to low, no mobile redirect but I do have a redirect on HTTP to HTTPS on all pages.
I also have IPv6 on although I doubt that would make any difference.

What I can off is that your application might be blocking the requests maybe, not CF.
Do you run custom code or is it an off the shelf solution like Wordpress?
And do you run it on AWS? EC2 instances or though an ELB?

I would suggest as your next step to check if you see any requests from Moz to your server
Just trigger a Crawl test on this link and wait a few hours, you will get the report by email
https://moz.com/researchtools/crawl-test

Let us know how it goes

Hi Rodrigo,

You need to be on CloudFlare Pro which is $20/month to get a dedicated IP address for SSL which will solve your problem.
You don't need to disable browser integrity but you might need to wait a few minutes until CloudFlare adds the IP to your domain. You can check it with a tool like https://www.whatsmydns.net/

Hi Roel,
I totally agree with you, but Moz seems to be really slow to move on this (it's been over 4 years since python released SNI support).
If you really want to use Moz I would strongly suggest just making the changes on your CDN instead of waiting for them to make a move, and you can always revert the setup once they release SNI support.
Google Cloud supports dedicated IP (they call it a VIP - virtual IP) and it costs $39/month.
You can find some info on how to setup here, I have never done it but seems pretty straight forward
https://cloud.google.com/appengine/docs/python/console/using-custom-domains-and-ssl

Guys,

I don't work for Moz and I I know it sucks that they don't support SNI, even if every major browser since IE6 does it, but it isn't a hard fix to add a dedicated IP address for SSL.
If you use AWS you can get CloudFront's dedicated IP for $600/month or you can add CloudFlare Pro for $20/month.
Every hosting company I know provides dedicated IP for SSL so the only issue here is the additional cost.
You guys are paying $100/month for Moz so might as well add another $20 (per domain) and forget about it
Happy to give some guidance with the setup if anyone isn't technical, you should also be able to switch the DNS with zero downtime.

Cheers,
Lucas

Hi David,

I did some more testing and it seems that the Crawler test was able to scan with the browser agent check on which is good news.
Moz will be indexing our site over the weekend and I can let you know then and confirm everything works as expected.

Cheers,
Lucas

So good news on my side, I was able to get the crawler test to work last night
My current setup is

• CloudFlare on Pro plan $20/month - you need this because it sets an static IP to your website
• On CF settings, disable BrowserAgent security checks - I still need to review this further but it seems CF blocks Moz's Rogerbot crawler because they cannot verify the user agent

I'll keep you posted on any new developments I have

Hi there,

We upgraded our site to CloudFlare Pro which supports all browsers that have TLS 1.0 and above - it gives the domain an static IP which means no SNI.
OpenSSL also connects to the domain without any issues but no luck in getting the crawler to work.
I also disabled Browser agent check security on CloudFlare which could be causing issues but that also didn't work.

Can anyone point me in the right direction?
Is there any way to get MOZ to crawl a site hosted on CloudFlare or is this a deadend?

If someone could provide a bit of code which I can use to test and debug the crawling request without having to use the crawling test tool that would be very helpful.

Cheers,
Lucas

Hi Roel,
I totally agree with you, but Moz seems to be really slow to move on this (it's been over 4 years since python released SNI support).
If you really want to use Moz I would strongly suggest just making the changes on your CDN instead of waiting for them to make a move, and you can always revert the setup once they release SNI support.
Google Cloud supports dedicated IP (they call it a VIP - virtual IP) and it costs $39/month.
You can find some info on how to setup here, I have never done it but seems pretty straight forward
https://cloud.google.com/appengine/docs/python/console/using-custom-domains-and-ssl

Hi there,

We upgraded our site to CloudFlare Pro which supports all browsers that have TLS 1.0 and above - it gives the domain an static IP which means no SNI.
OpenSSL also connects to the domain without any issues but no luck in getting the crawler to work.
I also disabled Browser agent check security on CloudFlare which could be causing issues but that also didn't work.

Can anyone point me in the right direction?
Is there any way to get MOZ to crawl a site hosted on CloudFlare or is this a deadend?

If someone could provide a bit of code which I can use to test and debug the crawling request without having to use the crawling test tool that would be very helpful.

Cheers,
Lucas

So good news on my side, I was able to get the crawler test to work last night
My current setup is

• CloudFlare on Pro plan $20/month - you need this because it sets an static IP to your website
• On CF settings, disable BrowserAgent security checks - I still need to review this further but it seems CF blocks Moz's Rogerbot crawler because they cannot verify the user agent

I'll keep you posted on any new developments I have

Hi David,

I did some more testing and it seems that the Crawler test was able to scan with the browser agent check on which is good news.
Moz will be indexing our site over the weekend and I can let you know then and confirm everything works as expected.

Cheers,
Lucas

Guys,

I don't work for Moz and I I know it sucks that they don't support SNI, even if every major browser since IE6 does it, but it isn't a hard fix to add a dedicated IP address for SSL.
If you use AWS you can get CloudFront's dedicated IP for $600/month or you can add CloudFlare Pro for $20/month.
Every hosting company I know provides dedicated IP for SSL so the only issue here is the additional cost.
You guys are paying $100/month for Moz so might as well add another $20 (per domain) and forget about it
Happy to give some guidance with the setup if anyone isn't technical, you should also be able to switch the DNS with zero downtime.

Cheers,
Lucas

Hey Jansen,

A few tips from me, hope it helps:
1. You don't need to pause CloudFlare completely, you can go into DNS and change the orange "cloud" to a grey one which will disable CF only on that record
2. Initially I thought about creating a new record for Moz which I could use for indexing (i.e. moz.yourdomain.com) but that will lose all search ranking records so not a good idea
3. Moz definitely works with CF + SSL, I have Moz it working on theright.fit domain, I reckon with a bit more help and testing we can get to the bottom of it, if you got time let's put together a complete list of settings and see where mine differs from yours
4. The domain has Mirrage ON, RocketLoader OFF and Minify ON (all options)