All services are now restored and stable. We are resolving this incident.
As per root cause, Microsoft has published their analysis: "We’ve determined the underlying cause. A backend cluster management workflow deployed a configuration change causing backend access to be blocked between a subset of Azure Storage clusters and compute resources in the Central US region. This resulted in the compute resources automatically restarting when connectivity was lost to virtual disks."
Posted Jul 18, 2024 - 23:10 EDT
Monitoring
Microsoft appears to be making substantial progress in restoring services. Our Azure SQL environment in Central US has come back online and all geo-replicated databases have caught up to their primaries.
As a result, we can now generate Custom Reports and OpenData should be operational as well.
We're still experiencing a bit of slowness here and there and will be keeping a close eye on things for a while.
Posted Jul 18, 2024 - 22:28 EDT
Update
Microsoft says they are currently applying mitigations, but we do not yet have an ETA for service restoration.
Clarifying our earlier notifications: Even though it is the Azure Central US region that is impacted, ALL customers of TrackAbout are presently unable to run Custom Reports and use OpenData regardless of that customer's global region. Those two TrackAbout services (Custom Reports and OpenData) have dependencies on Azure services operating in the Central US region.
TrackAbout is currently implementing disaster recovery procedures to migrate services to a different Azure region. We'll see who gets there first: us or Microsoft.
Posted Jul 18, 2024 - 21:25 EDT
Update
The Azure outage does seem to be confined to the Central US region, and not Global as Microsoft stated.
Microsoft hasn't released an update in over an hour. They have not indicated they understand the root cause.
We are continuing to monitor and hoping for some kind of update from Microsoft.
TrackAbout Custom Reports and OpenData remain unavailable for the time being.
Posted Jul 18, 2024 - 20:21 EDT
Update
Microsoft Azure has updated its status page to indicate they are having problems outside just the Central US region, calling it a "Global" issue.
Our services are operational (aside from the aforementioned Custom Reports and OpenData) at the moment. We're continuing to monitor the situation closely.
Posted Jul 18, 2024 - 19:32 EDT
Update
It appears several services in the Azure Central US region are down (storage, VMs, just about everything).
As it pertains to our OpenData service, if you have an open connection to the database, it may continue working. New connections to the database are mostly failing.
Custom Reports are unlikely to work until this is resolved.
Posted Jul 18, 2024 - 19:08 EDT
Identified
Starting at 6:13 PM Eastern US (22:13 UTC) on 18 Jul 2024, Azure has reported a failure in the Central US region in Azure SQL Database.
The impact for TrackAbout customers is that Custom Reports from the application web site may not load and new OpenData SQL connections may fail.
We are awaiting an update from Microsoft Azure on when service will be restored.
Posted Jul 18, 2024 - 18:53 EDT
This incident affected: Production Services (Custom Reports and OpenData).