Incident Library

IL-0332 Travis CI · 2018-03-13database

A test run against the production database truncated every table and travis-ci.com was down for about 5.5 hours

Many days earlier, a developer had used a terminal window in a tmux session to inspect production data, with DATABASEURL set to the production database. Travis CI's tooling and processes made it hard to connect to the read-only follower, so connecting to the primary with write access was a common shortcut. On 13 March 2018 the developer ran the test suite in that window, and the Database Cleaner gem truncated all tables in production; the query was blocked for about 10 minutes and ran at 12:14 UTC. Finding this took a day, using API logs and the source IP address of the query supplied by the upstream provider.

Responders left the user-facing applications running for about 30 minutes against the nearly empty database, which created duplicate user ids. All applications were then taken offline and a replacement database was provisioned from a snapshot taken that morning; provisioning finished earlier than first estimated, the database was reattached at 17:06 UTC, and builds for private repositories were running again at 17:13 UTC. travis-ci.com had been down for about 5.5 hours, and the build backlog on the Mac and sudo-enabled infrastructure took about another 3.5 hours to clear, by 20:59 UTC. About 15 minutes of data were lost. Users who had signed in during the 30 minutes were logged in as other users after the restore, and all affected tokens were revoked by 14:22 UTC on 14 March. Travis CI reviewed its logs, contacted every customer who was technically affected, found that some had potentially had build logs exposed, and advised them to rotate credentials stored on travis-ci.com and check their build logs for sensitive information. The cron scheduler had not been restarted after the restore, so scheduled cron jobs failed to trigger.

Severity
–
Scope
travis-ci.com non-operational for about 5.5 hours from 12:14 UTC and a build backlog for about 3.5 hours more; about 15 minutes of data lost; users who signed in during the first 30 minutes were later logged in as other users, and some build logs were potentially exposed; scheduled cron jobs failed to trigger after the restore
Users affected
New and existing travis-ci.com users; those who signed in during the 30 minutes after the truncation were logged in as wrong users
Region
–
Time to detect
–
Time to resolve
5h 30m

Symptoms

On 13 March 2018 at 12:14 UTC every table in travis-ci.com's production PostgreSQL database was truncated. Alerts fired right away, and customers reported missing account data; people who signed in saw blank user profiles.

The API kept running for about 30 minutes against the almost empty database. For each user who signed in during that time, travis-ci.com created a new user record, and because PostgreSQL does not reset id sequences on truncate, the new records took primary keys from the existing sequence. Users synced from GitHub got new records too, so both new and existing users were affected.

All applications were then taken offline while a replacement database was provisioned from a snapshot taken that morning. After the restore, people who had signed in during those 30 minutes found themselves logged in as the wrong users: the signed token in localStorage that served as their login credential matched user records created after the system restore. Scheduled cron jobs also failed to trigger, because the cron scheduler was not restarted after the restore and the alerts that would have shown this had not been turned back on.

Symptom Data lossService outage

Root cause

Root cause Operator error

Component Database
Contributing factor No safeguard on destructive changeHarmful manual action
Trigger
At 12:04 UTC a developer ran a test suite in an old tmux shell that still had a DATABASEURL environment variable pointing at the production database, and the Database Cleaner gem sent a truncate query to it. The query was blocked for about 10 minutes and executed at 12:14 UTC.

Detection

Alerts fired immediately after the truncation, and customers reported missing account data.

Remediation

Action items

Lessons

Developers connected to the production primary with write access because the tooling made the read-only follower hard to reach. While responding, Travis CI left user-facing applications running, which created duplicate user ids, and did not turn some alerts back on, which hid the stopped cron scheduler. Even the most experienced developers can make an inadvertent mistake that causes a large outage. The whole production database was recovered with only about 15 minutes of data loss.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Database among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.