The service has remained stable since the last update, and we still have more capacity coming online shortly.
We are meeting with Microsoft Azure engineers tomorrow to discuss the change in performance behavior we are seeing lately with Azure SQL Database elastic pools. The last few weeks we have seen some very anomalous behavior and we need to get to the bottom of it. We've had several back-and-forth exchanges over email and we're not getting the answers we need. We've escalated to having a call tomorrow and we'll continue to escalate until we get to a satisfactory answer.
We regret having to take the full outage tonight and we're sure we have some frustrated customers at the moment. We do apologize for the performance incidents of late and we're working hard to do better.
Posted Nov 14, 2023 - 05:10 EST
Monitoring
The service appears stable, although there is some elevated load for a couple big customers. We are bringing additional capacity online now which should help.
Posted Nov 14, 2023 - 04:38 EST
Update
We are now bringing services back online.
Posted Nov 14, 2023 - 04:09 EST
Update
We are going to have to take a full outage. We have too much backlog of work and the infrastructure cannot catch up. We'll report back as soon as possible.
Posted Nov 14, 2023 - 03:50 EST
Identified
We've made a change to the database tier and are waiting for it to complete to relieve the congestion.
Posted Nov 14, 2023 - 03:02 EST
Investigating
We are currently investigating a service outage.
Posted Nov 14, 2023 - 02:33 EST
This incident affected: Production Services (Production Application Web Site, Production TAMobile 6 Sync Services, TAMobile 7 iOS, TAMobile 7 Android).