This morning's incident during deployment was due to human error that occurred in composing a deployment package for our regular monthly Production update.
Our deployment packages are created on our Build Server. A member of the deployment team executes a command specifying a particular tag number that indicates the version of the code to be deployed. This person used the wrong tag, which resulted in the deployment package including database changes that were not intended to go out until the next release.
The net effect of the mistake was that some page requests and other operations were erroring out due to a mismatch between the web server code and the database code.
Upon realizing the mistake, we quickly reverted the database changes.
Going forward, we are instituting a peer-review process for the commands required to create a deployment build destined for the Production environment.
We apologize if you experienced any service disruption as a result of this incident.
Larry Silverman Chief Technology Officer TrackAbout Inc