Resolved -
All services are now restored and stable. We are resolving this incident.
As per root cause, Microsoft has published their analysis: "We’ve determined the underlying cause. A backend cluster management workflow deployed a configuration change causing backend access to be blocked between a subset of Azure Storage clusters and compute resources in the Central US region. This resulted in the compute resources automatically restarting when connectivity was lost to virtual disks."
Jul 18, 23:10 EDT
Monitoring -
Microsoft appears to be making substantial progress in restoring services. Our Azure SQL environment in Central US has come back online and all geo-replicated databases have caught up to their primaries.
As a result, we can now generate Custom Reports and OpenData should be operational as well.
We're still experiencing a bit of slowness here and there and will be keeping a close eye on things for a while.
Jul 18, 22:28 EDT
Update -
Microsoft says they are currently applying mitigations, but we do not yet have an ETA for service restoration.
Clarifying our earlier notifications: Even though it is the Azure Central US region that is impacted, ALL customers of TrackAbout are presently unable to run Custom Reports and use OpenData regardless of that customer's global region. Those two TrackAbout services (Custom Reports and OpenData) have dependencies on Azure services operating in the Central US region.
TrackAbout is currently implementing disaster recovery procedures to migrate services to a different Azure region. We'll see who gets there first: us or Microsoft.
Jul 18, 21:25 EDT
Update -
The Azure outage does seem to be confined to the Central US region, and not Global as Microsoft stated.
Microsoft hasn't released an update in over an hour. They have not indicated they understand the root cause.
We are continuing to monitor and hoping for some kind of update from Microsoft.
TrackAbout Custom Reports and OpenData remain unavailable for the time being.
Jul 18, 20:21 EDT
Update -
Microsoft Azure has updated its status page to indicate they are having problems outside just the Central US region, calling it a "Global" issue.
Our services are operational (aside from the aforementioned Custom Reports and OpenData) at the moment. We're continuing to monitor the situation closely.
Jul 18, 19:32 EDT
Update -
It appears several services in the Azure Central US region are down (storage, VMs, just about everything).
We are tracking the issue on Azure's status page here https://azure.status.microsoft/en-us/status/
We will update when we have more information.
As it pertains to our OpenData service, if you have an open connection to the database, it may continue working. New connections to the database are mostly failing.
Custom Reports are unlikely to work until this is resolved.
Jul 18, 19:08 EDT
Identified -
Starting at 6:13 PM Eastern US (22:13 UTC) on 18 Jul 2024, Azure has reported a failure in the Central US region in Azure SQL Database.
The impact for TrackAbout customers is that Custom Reports from the application web site may not load and new OpenData SQL connections may fail.
We are awaiting an update from Microsoft Azure on when service will be restored.
Jul 18, 18:53 EDT