TrackAbout System Status
All Systems Operational

About This Site

Sometimes things go wrong. On the rare occurrence they do, we want to be as transparent about our status as we can.

So if things aren't entirely green and operational, just know that it's not you. It's us. And we're working on it.

You can use this page to sign up for notifications about upcoming maintenance and real-time alerts about service issues.

Production Services Operational
90 days ago
99.99 % uptime
Today
Production Application Web Site ? Operational
90 days ago
99.98 % uptime
Today
Production TAMobile 6 Sync Services ? Operational
90 days ago
99.98 % uptime
Today
TAMobile 7 iOS ? Operational
90 days ago
99.98 % uptime
Today
TAMobile 7 Android Operational
90 days ago
99.98 % uptime
Today
Application-Sent Email Delivery ? Operational
90 days ago
100.0 % uptime
Today
Custom Reports and OpenData ? Operational
90 days ago
100.0 % uptime
Today
Production SFTP ? Operational
90 days ago
100.0 % uptime
Today
Production FTPS ? Operational
90 days ago
100.0 % uptime
Today
Customer Test Services Operational
90 days ago
99.99 % uptime
Today
Customer Test TAMobile 6 Sync Services ? Operational
90 days ago
100.0 % uptime
Today
Customer Test Application Web Site ? Operational
90 days ago
99.98 % uptime
Today
Marketing Operational
90 days ago
100.0 % uptime
Today
Corporate Web Site ? Operational
90 days ago
100.0 % uptime
Today
Customer Support Operational
90 days ago
100.0 % uptime
Today
Inbound Phone Service ? Operational
90 days ago
100.0 % uptime
Today
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Major outage
Partial outage
No downtime recorded on this day.
No data exists for this day.
had a major outage.
had a partial outage.
Production Uptime (via Pingdom) ?
Fetching
Past Incidents
Mar 25, 2023

No incidents reported today.

Mar 24, 2023

No incidents reported.

Mar 23, 2023

No incidents reported.

Mar 22, 2023

No incidents reported.

Mar 21, 2023
Resolved - This incident has been resolved.

We updated the deployment scripts that failed during the last deploy and redeployed.

Mar 21, 13:47 EDT
Investigating - The TrackAbout Customer-Facing Test environment (test.trackabout.com) is currently in a degraded state due to a failed code deploy. We're working to bring it back online ASAP.

A new method we created for deploying database schema changes to all customer databases failed due to the higher quantity of databases in the Test environment compared to our development and QA environments.

We know where the problem lies and we're working on a fix.

We're aware some of our customers use this environment for employee training.

Please know that as this is a Test environment, the test.trackabout.com site does not have the same Service Level Agreement (SLA) as our Production environment. Regardless, we're working to get it operational as quickly as possible.

Mar 21, 12:25 EDT
Mar 20, 2023

No incidents reported.

Mar 19, 2023

No incidents reported.

Mar 18, 2023

No incidents reported.

Mar 17, 2023

No incidents reported.

Mar 16, 2023

No incidents reported.

Mar 15, 2023

No incidents reported.

Mar 14, 2023
Postmortem - Read details
Mar 14, 15:52 EDT
Resolved - This incident has been resolved. We'll be publishing a post-mortem soon with our findings and improvements we'll be making going forward.
Mar 14, 11:02 EDT
Update - The system appears to have stabilized. We are monitoring.
Mar 14, 05:22 EDT
Update - We're continuing to see some slow performance here and there. We're currently thinking it's due to backlog of work from earlier. We're monitoring the situation and keeping an eye on individual high-load databases.
Mar 14, 05:04 EDT
Monitoring - We believe we have identified the root cause of the sudden worker process spike from 3% to 100% that caused the outage.

We have a separate team working on building a data warehouse/BI offering. They had scheduled a number of heavy data migrations for 1 AM Eastern Daylight Time, the exact time we started experiencing problems.

We found the data warehouse job and killed it. The system began recovering almost immediately.

We expect operations to return to normal shortly.

Mar 14, 04:04 EDT
Update - Azure SQL Elastic Pool transition is at 86% complete.

We've determined that at about 1:00 AM Eastern Daylight Time, our SQL pool worker percentage (the percentage of available worker processes in the SQL environment) went from 3% right up to 100% and stayed there. We're trying to find the cause of this very sudden spike in load.

When the workers in the SQL engine are exhausted, it cannot process further work until things calm down.

Increasing the size of our Azure SQL Elastic Pool as we have done increases the maximum number of workers.

Finding the cause of this spike is our highest priority.

Mar 14, 03:41 EDT
Update - While the Azure SQL Elastic Pool is migrating, we are seeing numerous database connection timeouts which indicate some of our customers are unable to use the application. The capacity migration is at 45% complete at this time. It started at 31 minutes past the hour. We hope to have operations restored in less than an hour.
Mar 14, 03:09 EDT
Identified - We've identified the Azure SQL Elastic Pool is again suffering poor performance under load. The pool is currently transitioning to increase power. This may result in periods of inaccessibility while the transition takes place.
Mar 14, 02:48 EDT
Investigating - We are investigating a report of slowness of the site. We are boosting capacity of our Azure SQL instance to compensate.
Mar 14, 02:37 EDT
Mar 13, 2023

No incidents reported.

Mar 12, 2023

No incidents reported.

Mar 11, 2023

No incidents reported.