On Thursday, September 14th, 2023 at approximately 4:12 PM, we received word from a single customer having login problems with the TrackAbout web site (but not the mobile apps). Their browser was reporting an error of type "NS_ERROR_NET_INADEQUATE_SECURITY" when attempting to log in.
After investigating, we could find no cause for such an error. No TrackAbout infrastructure changes had been made that day. We were left to conclude that this must be a local issue with the customer's infrastructure, most likely an HTTP proxy issue.
Several hours later, we received the same complaint from a second customer. With two disparate customers having the same issue, we turned our eyes to our new identity provider Auth0 (Okta) and opened a support ticket.
After some hours, Auth0 responded that they *had* made an infrastructure change without notification and that they had reverted the change. Their change was the cause of our customers' authentication issues.
This morning we received the following post-mortem report from Auth0:
---- "I want to start by apologizing for the impact this had on you and your clients, I can certainly understand why this would put you in a challenging position and I'm sorry that occurred. Our Engineering team provided a summary of the event that I've copied below:
On September 15, 2023, our Engineering team began to investigate a series of TLS cipher negotiation failures impacting our customers across multiple Private Cloud environments as well as our Public Cloud US region. Specifically, a subset of browsers began to experience errors when attempting to use our services through a custom domain.
The root cause of this issue was traced back to certain clients negotiating a cipher with our edge provider, then being rejected due to the cipher being on a banned list.
No further change is required by our customers and we do not expect this issue to recur following the rollback performed by our Engineering team. We sincerely apologize for any impact this had on you and your users.
I can add that the change was made to mitigate a separate certificate related issue that our Engineering team had identified as being a potential problem in the future. This is all of the information that has been provided about this event so far and I hope it's helpful for you and your team. Again I'm sorry for the position this put you in with your clients and please let me know if you have any follow up questions after reviewing." ----
As a preventative measure, we are adding change monitoring of the TLS/SSL certificates protecting our Auth0 authentication URLs. We need to know as quickly as possible if any change has been made to the configuration that Auth0 is managing. We are writing our own monitoring test as well as employing a popular TLS/SSL site quality checker from Qualsys SSL Labs.
Additionally, Auth0 is a relatively new third-party dependency for us, and a critical one at that, as it is the gateway through which all users authenticate.
Going forward, if we receive word of users having login problems, we will immediately open a ticket with Auth0.
Finally, I have let Auth0 know that making unannounced changes involving endpoint security has the potential to put their customers in an uncomfortable position and have strongly recommended that they publish notifications of changes.
Larry Silverman Chief Technology Officer TrackAbout, Inc.