Moz Pro, our web crawler, and sites that use SNI (804 HTTPS SSL) error

[fivefifty]

Hello:

Have a clarifying question: using this site:  https://globalsign.ssllabs.com/analyze  one of our sites got the following status:

HTTP Strict Transport Security (HSTS) with long duration deployed on this server.

and not this one:

This site works only in browsers with SNI support.

However, we recently received this error through Moz:

Error Code 804: HTTPS (SSL) Error Encountered
Your page requires an SSL security certificate to load (using HTTPS), but the Moz Crawler encountered an error when trying to load the certificate.

Please advise if the HTTP Strict Transport Security (HSTS) also causes the Error Code 804: HTTPS (SSL) Error Encountered in Moz?

Thank You

[DavidLee]

Anytime!

[etiennebeneke]

Brilliant David,

I appreciate the lengths you have gone to get more info on my issue. I will get the host to address this. Thanks very much!

Etienne

[DavidLee]

Hi Etienne

I checked the campaign you have with the 902 response and the issue is with your host blocking AWS IPs. I could not find any SSL or SNI errors for the domain.

You will want to reach out to your hosting provider to make sure they are not blocking any AWS IP ranges. Our IPs are dynamic so we will not be able to provide a range for you.

There are also other issues with the SSL configuration which will need to be looked at which could result in different errors once the IPs are unblocked.

http://www.screencast.com/t/b8qEdNah

You can run a test here for any SSL domain: https://globalsign.ssllabs.com/analyze.html

Hope this helps!

[jon.white]

Hello Etienne - sorry to hear this.

Have you checked in with help@moz.com yet? They might be able to fix your problem.

[etiennebeneke]

Hi Jon

Same problem this side, not sure if it's SNI. First we had 904 server error and then 902. Our site has not been crawled by RogerBot since 9 Oct when we switched from HTTP to HTTPS.

I am hoping for the best as I cannot use the full functionality of MOZ at the moment as we have one crawl error which completely keeps me in the dark as a SEO.

Many thanks
Etienne

[planetargon]

Do you have a rough ETA on the new crawler? Being without useful crawl data for several months has been painful

[TrueluxGroup]

Can you recommend any suitable alternatives for web crawling our site while your new crawler is in development.

We already use screaming frog but it would be handy to have something that flagged up any problems like MOZ did with our other sites.

Cheers

[AaronHemmelgarn]

OK great, thank you.

[DavidLee]

Yes the sign up link is in the very first post by Jon

The link is http://goo.gl/forms/LCvL9Ix8JDHfbAvr1

[AaronHemmelgarn]

Hi @David,

You mention a beta, is there someplace that we signup? I am a Moz Pro account holder and haven't seen anything offered or an e-mail.

Please provide steps for signup, as currently we have clients that we can't crawl and still paying for Moz.

[DavidLee]

Hello!

I can confirm your site does use SNI which can be checked at https://globalsign.ssllabs.com/analyze.html?d=www.honesthouse.co.uk&latest

If anyone else is unsure about your own site, you can use the above link, replace the URL and click on any server results which will confirm SNI status shown here: http://www.screencast.com/t/GLzjD7gjzjR

As for any workaround, other than disabling SNI completely, the quickest option to get crawled would be to sign up for the beta.

Jon will keep everyone posted with any new status and I can assure you this is our top priority feature we are working on.

[TrueluxGroup]

Hi Jon,

I think we might have the same problem.

Our site is https://www.honesthouse.co.uk/ and we're getting the 804 SSL error too

Is there anything we can do?

Cheers

[jon.white]

Hello Aaron!  Unfortunately I am unable to provide a date - mainly because if I did, I would be guessing, and I don't think that is helpful.  We used to commit dates to releases, and we stopped, simply because our software is too complex to estimate accurately.

I will say it is close, I sent an email update earlier this week to all who sign up for the beta, I have included this below, it has a little more color:

Thank you for signing up for the Moz Pro Site Crawl beta.  I wanted to give you all an update since it has taken us a little longer to get the beta ready for testing.  We were hoping to have it out here by MozCon (this week) but we are slightly behind.  In the interest of transparency:

• our current crawler does not support SNI (as you well know) and is simply too old and fragile to make meaningful updates to, so we decided in Q1 of this year to re-write our crawler
• the new re-written crawler will support SNI, be much faster and more reliable, and support many more configuration options inc. crawl speed, includes/excludes, UA (though some of these will not be available for the first beta)
• we will also be giving the UX and UI of the crawl a revamp in combination with this crawler
• whilst the SNI support works right now, we are ironing out some issues with the other parts of the crawler before we feel comfortable releasing it in beta
• we had hoped we would be able to show off the beta at MozCon this week but some recent changes have slowed us a little
• we will be offering a second free-trial for those who are not current customers and wish to try it again when it is ready

I appreciate your patience, we are very close and hope to have something soon.

[AaronHemmelgarn]

Do we have a beta timeframe yet? It's been 3 months since the announcement that SNI crawl ability is coming and still no release. I have a very large client that I can't crawl their site. As a development company I can understand that good code takes time, but 3 solid months and really a over a year is what we're really looking at that Moz has known this was an issue.

Please provide us (as customers) with a solid date that SNI will be supported. I don't think that's too much to ask.

[RobinGeall]

And a vote from me too Jon, I would need to this before converting from trial.

Good luck with it.

[John-Sharkey]

Just another vote to see this happen.  I use SNI on some sites and a must have to be able to use Moz.

As a side not, I am not seeing any Google crawl errors in using SNI, at least not in Google Search Console.

[jon.white]

Hi Phil - sorry you are seeing those errors.  I do not have a firm date, it will likely be in the next few weeks though.

[Globalgraphics]

Hi Jon, Do you have an update as to when the new crawler will be available. We have just started getting 804 HTTPS SSL error's for one of our sites.

[jon.white]

Thank you. I look forward to sharing something.

[matthewnmartin]

Great, we would like to participate as some results are better than nothing. We filled out the form so i suppose we wait.

[jon.white]

Thanks for your response.

Firstly, I empathize with your decision to cancel, and I hope we can win you back in the future.

In terms of why a customer might participate. Many of our customers who have SNI problems are still Moz customers, and still use our Rankings, Links and Keyword Research tools.  In this case, they would also be able to get some crawl data (albeit in a beta form) as a temporary fix whilst we finish the crawler.  We had already planned to launch a beta with select customers, we think this might also help us help some customers who are having struggles with our product.  But I realize this route is not for everyone.

Please let me know if there is any other feedback you guys have about Moz.  jon@moz.com

[matthewnmartin]

Thank you for the update. We have several sites that cannot be crawled and after a lot of investigation have concluded we would need to cancel Moz. I'm glad this is being released and submitted the form to be contacted regarding the Beta Testing.

I am curious though, what is in it for us? As of now we are paying for a service that does not work. I seriously doubt we will be interested in continuing to pay for a service just to be a test dummy (which one of the requirements is a paid account, unless I am misunderstanding).

Either way, we are excited about the solution and interested in learning more.

Thanks.