The root cause of the performance issue was traced to ... well, let's just say it's not a good idea to attach 24,747,737 Delivered Not-Scanned assets on a single record and then attempt to view the Record Detail page where each NS asset is displayed individually.
We're going to look into enforcing a cap on Not-Scanned quantity attached to a single record and see if we can add short-circuit protection to the Record Detail/Summary web page in case a large quantity of Not-Scanned assets do get attached to a record.
Posted Feb 01, 2024 - 03:05 EST
Monitoring
The system has stabilized. We are monitoring.
Posted Feb 01, 2024 - 02:17 EST
Investigating
We are currently investigating a performance issue with the site. There appears to be a customer workload that is causing extreme CPU consumption and server RAM growth. We're scaling the environment to accommodate while we trace the root cause.
Posted Feb 01, 2024 - 02:08 EST
This incident affected: Production Services (Production Application Web Site, Production TAMobile 6 Sync Services, TAMobile 7 iOS, TAMobile 7 Android).