Incident Library
Applied filters Root causeOperator errorComponentDatabaseOrganisationTravis CIClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0332 A test run against the production database truncated every table and travis-ci.com was down for about 5.5 hours
    Travis CI2018
    Symptoms

    On 13 March 2018 at 12:14 UTC every table in travis-ci.com's production PostgreSQL database was truncated. Alerts fired right away, and customers reported missing account data; people who signed in saw blank user profiles.

    Data lossService outage
    Trigger

    At 12:04 UTC a developer ran a test suite in an old tmux shell that still had a DATABASEURL environment variable pointing at the production database, and the Database Cleaner gem sent a truncate query to it. The query was blocked for about 10 minutes and executed at 12:14 UTC.

    Root cause
    Operator error
    Component
    Database
    Contributing factor
    No safeguard on destructive changeHarmful manual action
    Detection

    Alerts fired immediately after the truncation, and customers reported missing account data.

    Remediation
    • Took all applications offline
    • Provisioned a replacement database from a snapshot taken that morning
    • Reattached the restored database and brought platform services back up
    • Worked through the backlog of builds until it cleared
    • Revoked all affected login tokens by 14:22 UTC on 14 March
    • Contacted affected customers and advised them to rotate stored credentials and check build logs
    Action items
    • Revoke the truncate permission on the databases
    • Make the internal spechelpers check for the DATABASEURL environment variable
    • Make the developer shell prompt warn when a DATABASE_URL is set
    • Submit a pull request to the Database Cleaner gem guarding against a remote DATABASE_URL
    • Create an alias for the follower database so it is easy to connect to for work against production data
    • Automate database failover and maintenance to cut manual recovery steps
    Lessons

    Developers connected to the production primary with write access because the tooling made the read-only follower hard to reach. While responding, Travis CI left user-facing applications running, which created duplicate user ids, and did not turn some alerts back on, which hid the stopped cron scheduler. Even the most experienced developers can make an inadvertent mistake that causes a large outage. The whole production database was recovered with only about 15 minutes of data loss.

    View case
Check your cluster for this

This case involves Database. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Operator error1 case

Component
Database
Contributing factor
Harmful manual actionNo safeguard on destructive change
Organisation
Travis CI
  1. IL-0332 A test run against the production database truncated every table and travis-ci.com was down for about 5.5 hours
    Travis CI2018