I have recently switzed to SSL on ramxpert.dk, now Moz is not able to crawl the site anymore

[WebBoost]

Hi Moz

I have recently switzed to SSL on ramxpert.dk, now Moz is not able to crawl anymore.

When I look at the crawl report there is just one fault displayed:

"403 : Received 403 (Forbidden) error response for page."

How can i solve this issue asap?

I was not aware that there would be any issues with moz when activating a SSL certificate on the site.

The campaign with the problem is: ramxpert.dk.

Regards
Jesper Nielsen

[jocameron]

Hi Jesper,

Jo here from the Moz support team.

As WebBoost has noted our crawler isn't compatible with SNI, but the good news is that our team has been working hard on a new crawler which is in Beta right now and can handle SNI. If you're interested in being a part of the beta testing, the link to the signup page is here: http://goo.gl/forms/LCvL9Ix8JDHfbAvr1. Once you fill out that form, you will automatically be added into the Beta during the next round.

We will be rolling the new crawler out to all of our users once it's out of it's Beta phase, but I don't have an exact timeframe of when that will be at this time as it's still being worked on.

You're welcome to reach out to us at help@moz.com if you get stuck :]

Cheers!
Jo

[WebBoost]

Thanks BeadedSEO

It seems through that the issue is that ramxpert.dk uses SNI, which isn't supported by the moz crawler yet

I've decided to sign up for the beta version of the new crawler, which I found on the link below. Hopefully this will work while they work on the final version.

https://moz.com/community/q/moz-pro-our-web-crawler-and-sites-that-use-sni

[mcncl]

Have you tried updating your robots.txt file to use this;

User-agent: rogerbot

Disallow:

User-agent: dotbot

Disallow:

It's best practise to Disalow: none, rather than Allow: /

I've tried crawling your site with SEO Spider and ran in to no issues.