SSL prompts on certain devices only - after Firewall recovery

[achalmers 2]

pfSense / HAProxy failure.  Full rebuild and recovery.  Re-issued / Renewed my reverse proxy certs via ACME and Lets Encrypt.  Now only certain devices are having trouble accessing Emby.  Android devices are prompting to accept the SSL certificate.  Roku devices completely fail to login / no prompt or anything.  Just spinnging.   IOS devices login fine no issues and no cert prompt.  Windows PC's login fine no prompt and inspecting the certificate everything is valid / current. 

I tried deleting cache and all app data for Emby on my android device and re-logging in.  Same issue.  Prompting for SSL certificate acceptance.  Any ideas?  Thanks!

[Luke 37066]

Hi, it sounds like they're all rejecting your certificate, no? The only reason you get a prompt on android is because we have the ability to allow you to override that. On other platforms we don't though.

[Q-Droid 643]

When things are inconsistent upon cert renewal there's a chance you left out the chain/intermediate certs from the configuration. Make sure you included the cert.pem + chain.pem or the fullchain.pem for the reverse proxy.

 

[achalmers 2]

On 11/30/2022 at 6:30 PM, Luke said:

Hi, it sounds like they're all rejecting your certificate, no? The only reason you get a prompt on android is because we have the ability to allow you to override that. On other platforms we don't though.

I dont think iPhone, or Windows devices are rejecting the certs.  They are not prompting, and when inspecting the cert chain via Windows browser everything looks fine and these clients can access Emby without issue

 

When the firewall failed I loaded onto a new RAID1 array, installed the software, restored a backup, and went into ACME service and manually told them to renew.  

 

Certs stuff confuses the hell out of me sometimes.  I'm definitely no expert.  

Edited December 2, 2022 by achalmers
Wording

[achalmers 2]

On 11/30/2022 at 7:01 PM, Q-Droid said:

When things are inconsistent upon cert renewal there's a chance you left out the chain/intermediate certs from the configuration. Make sure you included the cert.pem + chain.pem or the fullchain.pem for the reverse proxy.

 

ok so I think you may be right.  I looked at the intermediate cert serial number (R3) and on the Windows device the Serial number is different than on my Android device.  Windows device is happy, Android is not.  How can the intermediate cert be different when both clients access the reverse proxy in the same way.  How can the reverse proxy be giving one a different intermediate cert ? 

[achalmers 2]

I got it figured out.  Thanks for the help you definitley pointed me in the right direction !  Looks like after recovery the HA Proxy backend was not set to use the CA as part of the certificate profile or something.  What has me baffled is how IOS and Windows clients were getting the CA but Android wasnt.... ?      Changed the CA field here from "None" to the Intermediate

[Q-Droid 643]

The trust/key stores are often different across devices. Even on the same devices different key stores may be used. For example some browsers and apps use their own while others use the system's key store. Then you run into cases where the key store includes the root and intermediate certs for some CAs but only the root for others. Add the somewhat recent intermediate cert rotation from Letsencrypt and the ability to validate the chain/root becomes hit or miss when not enough "links" are presented during the validation exchange.

In your case HAproxy was likely only presenting the server cert (depth=0) which some devices could handle because they had the intermediate in their store, but others didn't. Including the chain/intermediate in the config lets your server present a more complete chain (depth=1 or more depending on CA) giving the clients all of the "links" needed to validate the cert.