IL-0577 Ficilcom · 2026-09-25 (published)database
Ficilcom's production Neon PostgreSQL database was reset by prisma migrate reset meant for the local database
Ficilcom builds its product with Turborepo, Prisma and Neon (PostgreSQL). The database code lives in packages/database, and the same Prisma commands run against whatever the DATABASE_URL in that folder's .env points to, local or production; nothing guarded against this, and nothing checked that the URL was localhost before migrate reset. Before the holidays the engineer had run a migration against production and left the production DATABASE_URL in packages/database/.env. After the holidays the engineer did not remember this and ran the reset without checking, which reset the production database.
The engineer found Neon's Backup & Restore feature and used it to restore the database. On the Launch plan in use, the history window is 1 day by default and up to 7 days, and the history reached back one day, so the data could be recovered. The service was stopped for about an hour. The service was in its early launch phase with one customer's data, and access logs showed almost no impact.
- Severity
- –
- Scope
- The production database was reset and the service stopped for about an hour
- Users affected
- One customer's data was in the database; access logs showed almost no impact
- Region
- –
- Time to detect
- –
- Time to resolve
- 1h
Symptoms
On the first working day after the Silver Week holidays, an engineer ran prisma migrate reset and then loaded seed data, intending to clean the local database before resuming work. The seed data did not seem to be there, and the engineer first suspected that the coding agent had skipped the step. The reset had in fact run against the production database.
Symptom Data loss
Root cause
Root cause Operator error
- Trigger
- The engineer ran prisma migrate reset believing it targeted the local database, while packages/database/.env still held the production DATABASE_URL left there from a production migration before the holidays.
Detection
After the reset the engineer saw that the seed data did not seem to be present and realised the reset had hit production.
Remediation
- Restored the production database with Neon's Backup & Restore
Action items
- Make local scripts check DATABASE_URL and refuse to run destructive commands against anything but localhost or 127.0.0.1
- Stop writing production connection strings in local .env files
- Run production operations through CI or a secret manager
- Check local env files for production hosts such as *.neon.tech
- Check the Backup & Restore history window of the plan in use, and extend it or add separate backups if it is not enough
- Before holidays or long breaks, remove the production URL from local env files
- When stopping work midway, record how far it got in an issue or memo
- Add rules that forbid coding agents from running destructive database commands unless the user asks in the current message
Lessons
Memory after a long holiday cannot be trusted, so the state of unfinished work has to be written down instead. A destructive command should refuse to run when its target is not clearly local. That a backup feature exists does not mean the plan's history window is long enough, and the damage was small this time only because the service happened to be at an early stage.
Sources
References
Related cases
IL-0576Every dataset in CareNet's production BigQuery data warehouse was deleted, and UNDROP SCHEMA could not restore themCareNet · 2025same root cause · shared Database
IL-0242Both PostgreSQL servers behind matrix.org were lost during a disk expansion, and the homeserver was down for 24 hoursMatrix.org Foundation · 2025same root cause · shared Database
IL-0582Jobs piled up in Kanmu's PostgreSQL-based qg job queue on Aurora as dequeue slowed and dead tuples kept growingKanmu · 2024same root cause · shared Database
IL-0556A transaction left open during database maintenance stopped uploads and then downloads on Storj's US1 satelliteStorj · 2024same root cause · shared Database
IL-0531Azure Database for MySQL Single Server and MariaDB in West US rejected every new connection for a batch of serversMicrosoft Azure · 2024same root cause · shared Database
IL-0565Cookie Run: Kingdom's CockroachDB cluster failed when 16 of 24 nodes lost their filesystems, causing a 36-hour outageDevsisters · 2022same root cause · shared Database
IL-0207A routine task run from an engineer's local environment switched to production and deleted GitLab.com's Consul release, so Patroni lost Consul and GitLab.com returned 5xx errors to all trafficGitLab · 2022same root cause · shared Database
IL-0332A test run against the production database truncated every table and travis-ci.com was down for about 5.5 hoursTravis CI · 2018same root cause · shared Database
IL-0431During Mackerel's move to AWS, PostgreSQL failed over to a replica that had stopped replicating, losing an hour of dataHatena · 2017same root cause · shared Database
IL-0506UC Berkeley DataHub logins failed and students got empty home directories after the JupyterHub database was overwrittenUC Berkeley · 2017same root cause · shared Database
IL-0218GitLab.com's primary PostgreSQL data was wiped by mistake and, with no working backup, the site was down about 18 hoursGitLab · 2017same root cause · shared Database
IL-0573A developer's mistake dropped nine main tables of the Baemin billing MariaDB database and payments briefly stoppedWoowa Brothers · 2016same root cause · shared Database
IL-0319Dropping databases on the MySQL replica mysql2 also dropped them on mysql1, as the pair actually ran master-masterOregon State University Open Source Lab · 2015same root cause · shared Database
IL-0337EBS volumes in one US East Availability Zone got stuck after a network change cut EBS nodes off from their replicasAmazon Web Services · 2011same root cause · shared Database
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0577
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0577/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.