Incident Library
Applied filters FactorNo safeguard on destructive changeOrganisationFicilcomClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. Symptoms

    On the first working day after the Silver Week holidays, an engineer ran prisma migrate reset and then loaded seed data, intending to clean the local database before resuming work. The seed data did not seem to be there, and the engineer first suspected that the coding agent had skipped the step. The reset had in fact run against the production database.

    Trigger

    The engineer ran prisma migrate reset believing it targeted the local database, while packages/database/.env still held the production DATABASE_URL left there from a production migration before the holidays.

    Root cause
    Component
    Contributing factor
    Detection

    After the reset the engineer saw that the seed data did not seem to be present and realised the reset had hit production.

    Remediation
    • Restored the production database with Neon's Backup & Restore
    Action items
    • Make local scripts check DATABASE_URL and refuse to run destructive commands against anything but localhost or 127.0.0.1
    • Stop writing production connection strings in local .env files
    • Run production operations through CI or a secret manager
    • Check local env files for production hosts such as *.neon.tech
    • Check the Backup & Restore history window of the plan in use, and extend it or add separate backups if it is not enough
    • Before holidays or long breaks, remove the production URL from local env files
    • When stopping work midway, record how far it got in an issue or memo
    • Add rules that forbid coding agents from running destructive database commands unless the user asks in the current message
    Lessons

    Memory after a long holiday cannot be trusted, so the state of unfinished work has to be written down instead. A destructive command should refuse to run when its target is not clearly local. That a backup feature exists does not mean the plan's history window is long enough, and the damage was small this time only because the service happened to be at an early stage.

Check your cluster for this

This case involves Database. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

By root cause1