Moz Pro, our web crawler, and sites that use SNI (804 HTTPS SSL) error

[jon.white]

TL;DR

• Many sites and CDN’s across the internet use SNI to help keep pages and resources secure

• Right now, Moz Pro is unable to crawl sites that use SNI, affecting these customers (you will see an 804 HTTPS SSL error if you are affected)

• This is being fixed, and we are almost ready to provide beta access to SNI customers

• If you have an SNI site, are a Moz Pro customer, and wish to be a part of this beta - sign up here http://goo.gl/forms/LCvL9Ix8JDHfbAvr1

Detail

SNI (Server Name Identification) is an extension to the TLS security protocol that allows for multiple HTTPS websites to be served from the same IP address without requiring them all to use the same certificate.  This was always possible using wildcard certificates, but these are often impractical, since they are restrictive, and require a webmaster to know all of the hostnames to be served at the time of requesting the certificate.  With SNI, a webmaster can issue a single certificate and support multiple (and changing) hostnames.

SNI has become increasingly popular over the last couple of years thanks to:

• Increase in mainstream browser support

• Its flexibility for usage in CDNs, like Cloudflare

Unfortunately the Moz Pro web crawler has not kept up to date with this trend, and is at present unable to crawl sites that use SNI (the symptom is an 804 HTTPS SSL error).  This affects customers with SNI hosted sites, who miss out on the value of Moz Pro’s Site Crawl tools.  Whilst it might seem a simple fix to support SNI, our crawler software uses libraries that are unable to support the technology without a significant re-write.

However, I am happy to say that this re-write is in process, and almost ready for testing.  Behind the scenes we have been working on a major overhaul of our crawl technology which you are going to start to see across many of our products. In fact, Moz Content adopted the new crawler this week, resulting in much faster crawls and support for SNI.

Moz Pro is not ready for full implementation yet. That said, we are pretty close to being able to beta test with customers.  For that reason we would like to invite any customers with SNI sites into our closed Beta that will likely start some time in August.

Some notes about the beta:

• Initially, it will be a little behind where the current crawl tools are in terms of features and configuration. That said, we will be iterating fast and beta customers will get automatic access to all new features

• As with any beta there will be bugs and potential data outages - I want to be transparent about this up-front

• We will be looking for feedback throughout the beta, and we would love to have our customers help shape the future of our software

• You must be a paid Moz Pro subscriber (or in a free trial)

If you are interested please sign up here - http://goo.gl/forms/LCvL9Ix8JDHfbAvr1. Questions and comments welcome below!

[Christy-Correll]

UPDATE!

As of June 2017, this issue has been resolved. That's right, Moz Pro is now able to crawl sites that use SNI. For more details, please see https://moz.com/community/q/moz-crawler-fully-supports-sni-sites. You may also leave any questions and comments you have about this update there.

Thanks so much for your patience, ya'll!

[jon.white]

Hello! When did you request access? Can you email me directly at jon@moz.com

[omegadigital]

Hi Moz,

It is May 2017 and the issue still hasn't been resolved. I have a free trial account and the client used in the console has this error. I noticed there is an issue with SSL comodo users.

Any updates? I have requested BETA access to the new crawler.

[Robert-Jordan]

Hey Hi,

Even the same issue with my website https://www.medicoreach.com please kindly help.

Thanks in advance.

[jon.white]

Hi Dean. Sorry for the frustration. It is a huge priority and is currently fixed / working in beta.  Send the email address you use to log in to Moz Pro to jon@moz.com and I will add you.

[Reunionvacationhomes]

We have SNI errors for our Streamline sites and we use CloudFlare. Moz, how big of a priority is it for you to fix this?

How would we opt-in to the beta?

[fabfrug]

Jon, If we are not in the Beta test and are currently using CloudFlare and getting the 804 Error code in our crawl results. Should we just ignore this notification or is there anything else we should be concerned about?

[vital_hike]

Hi Jon, Just filled in the request form and you say we will be added in the next batch. Will we receive an email when we are added or will we need to keep checking. If you could let me know that would be appreciated.....save me checking all the time.

[jon.white]

Hello!  I looked it up, and it looks like you signed up for the beta 5 days ago. We add new beta customers once per week, which means your account is in the batch to be added this week.  I am really sorry for your frustration and appreciate your patience.

[jmsdonline]

@Darren, no.. it's not there and I would be mindful to not post if it was present.

@Jon, thanks for the response. I signed up with admin[at]jmsolodesigns.com - this is both for login as well as the email submitted on the form.

[O-IO]

You might want to go back and look. It has been working in the SiteCrawl v2 [BETA] tab of your project in the ProTools. Head there and see if you see the tab.

[jon.white]

Hi - which mail address did you use to sign up for the beta?  And which email address do you use to log in to Moz Pro?

[jmsdonline]

It's unfortunate that I have been following the progress of this for almost a year and every time it seems to be the same response from the Moz team... "Coming Soon". Well, I have applied for the beta program as all of my clients are using SNI via Let's Encrypt and some on CDN providers. It makes the price point almost NOT WORTH IT, nor does it instill much confidence to a potential customer when this issue has been bounding around the internet for almost a year with little to no head way.

Hopefully we get a response to this (the beta access) soon as I can not possibly justify spending the money on a service that other FREE tools can do. The one thing that would make this worth it is the ability to actually crawl my clients sites under one roof... which cannot be done atm.

[dylanbdg]

Submitted for beta. I'm still on the free trial and so far IN LOVE with Moz tools - but we have 60+ clients, many of which are on GoDaddy Managed WordPress hosting (I know, I know...) and we are recommending SSL to every one of them. I will be unable to justify expense of Moz if SNI is not supported - very much looking forward to this implementation! Thanks Moz crew.

[griffinassoc]

Having 804 errors on a site that is hosted with WPEngine and is using CloudFlare. I've submitted for the beta.

[johnhamman]

I am having the 804 error on the crawl for my sites too. Both hosted on wpengine.com and using cloudflare.com. I submitted to the beta.

[DavidLee]

Anytime!

[fying]

Thanks, David!

[DavidLee]

Hi Kristin

I can confirm your issue is not related to SNI or the new beta crawler as your site does not use SNI.

Outside of SNI and intermediate certificate issues, 803/804 responses will mostly be temporary at the time of the request to reach a page, which is very difficult to isolate unless you see the same URLs appear in each new report.

I wouldn't worry about the spike until the next crawl occurs where you will want to look for re-appearing URLs which will then help you isolate possible issues with your SSL configuration.

Export the CSV for the current URLs to compare with next weeks crawl.

Hope this helps

[fying]

Hi, Jon-

My site (https://www.paloaltou.edu) never encountered 803 or 804 issues until yesterday's report- which all of a sudden flagged 100 pages for this!

Did the opening of the beta crawler on 1/30/17 have any affect on the live version of the crawler that could've caused pages to now get flagged as 803/804 issues?

Thanks,

Kristin

[jon.white]

Hi Naomi.

Please can you email me the email address you use to log in to Moz Pro?  You can get me at jon@moz.com.  I will get you manually added.

Thanks,

Jon

[Naomi_AE]

Hi Jon,

Any news on my above message? If I cant crawl my site i'm going to have to cancel. which I don't want to do.

[TheJayeCowle]

Thanks Jon, just signed up!

[Naomi_AE]

Hi Jon,

Thanks for the reply. I don' see the option. If I can be added that would be great.

[planetargon]

Update: I signed out then back in... now see the Site Crawl v2.

[planetargon]

Thanks, Jon. I don't see that when I log in and choose a campaign. Do I need to sign out/back in?

[jon.white]

To find the beta, log into Moz Pro and choose a campaign.  You will see a link in the left navigation.

(missed that first time around so added it to the response)

[planetargon]

Where would we see a beta invite?

[jon.white]

Hello all,

Happy to be able to share that the beta is now open.  If you filled in the form above then you will have access to the beta as of today.  If not, please add your information and we will be adding new participants weekly.

Thanks for your patience,

Jon

[jon.white]

Hello!

Yes the beta is now open and I am happy to be able to add you. Please complete this form and we will add new users weekly.  https://docs.google.com/forms/d/e/1FAIpQLSeaJFsouoRBGF--pEK1MYfWJOhq6KU7NIGu6m2ZpNLnSXgsUA/viewform?c=0&w=1

To find the beta, log into Moz Pro and choose a campaign.  You will see a link in the left navigation.

Thanks,

Jon

[jon.white]

Happy to say we added all survey respondents to the beta today.  Let me know if you don't see the option?

[Naomi_AE]

Hi,
My site us having an SNI issue too. What is the update on the BETA test platform?

[TheJayeCowle]

Hi Jon,

I know you're kind of under the line of fire here but I'm getting this error too and have used that test to test the site where I'm getting this error and it looks like it's the SNI certificate indeed.

Are there any more spaces to be a part of the beta testing? also, would this affect all my campaigns or will you just create a test account with that campaign where I'm getting the error?

I wouldn't want to risk transferring all my clients' campaigns to a beta test platform.

Hope that makes sense.

I'm looking forward to hearing from you!

Mariano

[CroyalDK]

Hi,

Really need an update on this before boarding new clients??

[RCVanPoppel]

Hi Jon,

Thank you for the updates. We've added SSL to our website in December, as a result we've now also seen the SNI issue. Is the SNI also affecting DA and PA in site explorer? Our previous DA and PA were when the site did not have the SNI issue. Just want to know if we can compare them like for like now we're on SNI. Thanks in advance.

[jon.white]

Hello!  Very sorry for the delay in response here.

We are planning to open up the beta to all SNI customers beginning next week.  You will get an email to the email address you use to login.  We really appreciate your patience - thank you again.

[jon.white]

Hello!  Very sorry for the delay in response here.

We are planning to open up the beta to all SNI customers beginning next week.  You will get an email to the email address you use to login.  We really appreciate your patience - thank you again.

[Senson]

Would love an update. Been waiting for Moz to fix the SNI problem for what seems like ages.

[CroyalDK]

Any news on the SNI fix?

[IWaldron]

Any update to this estimate? Anxiously waiting...

[spadedesign]

We have yet to see a fix for this issue or receive any word of trying the new software.

We are going on 5 or 6 months.

[John-Sharkey]

Absolutely... thanks so much for your work on this!

Cheers,
John

[jon.white]

We are testing in-house beginning next week, I don't have a sense how long this testing cycle is going to take, but this is the final stage before we launch it.  Thanks for your patience.

[John-Sharkey]

Hello Jon,

Any update on the new SNI fix, or the beta crawler?  Maybe a Xmas present??

Kind regards,
John

[etiennebeneke]

Many thanks David for your effort. I will feed this information through to the server guys.
Etienne

[DavidLee]

Hi Etienne

There is a public 3rd party tool which also fails to connect which is hosted on AWS:

https://www.hurl.it/

If you run a GET request to this URL you will see a time out. Your admins can use this tool to quickly isolate any issues as this timeout is the same we are running into.

The SSL issues reported may not block normal browser connections but possible non-browser user-agents which would be another bot or server. We do not have any requirements or TLS, 1.0 and up is supported.

Hope this helps!

[fivefifty]

Thanks for the clarification.

[etiennebeneke]

Hi again David

I forwarded your response below to the server admin folks and they wanted me to check with you re this:

1. Perhaps MOZ can do a simple telnet test to www.ooba.co.za on port 443 to see if they can connect at the most basic level.

2. The SSL issues they are referring to are not very clear to me, I know we support SSL3, which is insecure but shouldn’t prevent anyone browsing the website. We also don’t support TLS 1.1 or 1.2, only TLS 1.0, do they have any specific requirements regarding TLS support?

I have to admit that most of this is rather Greek to me.

Thanks David

Etienne

[DavidLee]

Hi Ryan

I don't believe HSTS would prevent issues or this would apply to all https pages on every crawl report. For 804 response for any single page you want to monitor for the same URL appearing in the crawl report every week. Outside of SNI 804s will commonly be temporary issues.

If the same URL appears week to week with an 804 you would have to involve your web dev/host to further investigate.