We made an emergency change to the malfunctioning Azure SQL elastic pool, moving it to a higher performance tier, in an attempt to resolve the problem. This worked and connections are now able to be established to the affected customer databases.
We've been contacted by Azure engineering. They confirmed that a database component in Azure failed and they are performing a root cause analysis as to why.
There has been no data-loss as a result of this incident. The databases in question were read-only copies of production data, used for failover and reporting purposes only. The database replication is now catching up on delayed replicated transactions.
We are marking this incident Resolved as functionality has been restored.
Posted Sep 27, 2021 - 09:38 EDT
Investigating
The Custom Reports section of the web site and OpenData access are being impacted by a partial outage of a pool of read-only Azure SQL geo-replicated databases. These read-only replicas have become non-responsive.
We are investigating and will report more when we can. We have opened a service ticket with Microsoft Azure to investigate.
Posted Sep 27, 2021 - 08:59 EDT
This incident affected: Production Services (Production Application Web Site).