Issues resulting from current Production update
Incident Report for TrackAbout System
Postmortem

This morning's incident during deployment was due to human error that occurred in composing a deployment package for our regular monthly Production update.

Our deployment packages are created on our Build Server. A member of the deployment team executes a command specifying a particular tag number that indicates the version of the code to be deployed. This person used the wrong tag, which resulted in the deployment package including database changes that were not intended to go out until the next release.

The net effect of the mistake was that some page requests and other operations were erroring out due to a mismatch between the web server code and the database code.

Upon realizing the mistake, we quickly reverted the database changes.

Going forward, we are instituting a peer-review process for the commands required to create a deployment build destined for the Production environment.

We apologize if you experienced any service disruption as a result of this incident.

Larry Silverman Chief Technology Officer TrackAbout Inc

Posted Feb 21, 2017 - 13:41 EST

Resolved
All issues pertaining to the Production update have now been resolved.
Posted Feb 21, 2017 - 12:38 EST
Identified
We've identified the issue and working to restore normal function.

Symptoms users may have noticed include:

- Failures importing invoices or orders
- Failures while editing customers in the web site.
Posted Feb 21, 2017 - 11:54 EST
Investigating
We are experiencing some issues with the Production code update that is currently underway. We are writing to let you know we are aware and will update shortly with more information.
Posted Feb 21, 2017 - 11:00 EST