Every dataset in CareNet's production BigQuery data warehouse was deleted, and UNDROP SCHEMA could not restore them

Symptoms

On a day in August 2025, at 10:41, CareNet's system development division posted a first report on Slack that a failure was occurring in BigQuery, the production data warehouse. At 11:47 the division's investigation found that the production datasets had been deleted by a service account belonging to the AI technology development team.

The data warehouse is used for data analysis and is not connected to user-facing services; several departments share a single Google Cloud project to collect their data in it. At 11:00 the AI technology development team talked about the failure in a meeting but did not suspect its own team and only watched.

At 12:44 the system development division reported that an UNDROP SCHEMA query, the BigQuery feature for restoring a deleted dataset, did not work. The deleting process had created a new dataset with the same name right after deleting each one, and BigQuery appeared to treat the dataset to be restored as not existing.

Symptom Data loss

Root cause

Trigger

Test code that was being modified contained a step that resets test data by deleting a dataset and recreating it with the same name. Because of a configuration mistake, the step ran against production and deleted and recreated every production dataset.

Detection

The system development division posted the first report of a BigQuery failure on Slack at 10:41, and its investigation found at 11:47 that the AI technology development team's service account had deleted the production datasets.

Remediation

• Restored the tables one at a time with bq cp in Cloud Shell from BigQuery time travel, to a point before the incident
• Recreated views and table functions, which bq cp cannot restore, from the creation queries kept in BigQuery saved queries

Action items

• Enforce controls so that test code cannot access the production environment
• Remove the excessive permissions granted to the service account
• Keep dataset and table information as managed metadata, because restoring with bq cp requires knowing the table names
• Keep the creation queries of views and table functions in a place such as GitHub, because bq cp cannot restore them
• Consider how BigQuery should be backed up

Lessons

BigQuery time travel with bq cp recovered most of the data, but only because the table names were known, and it does not bring back views or table functions. Test code should not be able to reach production, and service accounts should not hold more permissions than they need.

Sources

• https://zenn.dev/carenet/articles/recovering-from-bigquery-dataset-deletion · blog · archived copy

Related cases

• IL-0577 Ficilcom's production Neon PostgreSQL database was reset by prisma migrate reset meant for the local databaseFicilcom · 2026

  same root cause · shared Database

• IL-0242 Both PostgreSQL servers behind matrix.org were lost during a disk expansion, and the homeserver was down for 24 hoursMatrix.org Foundation · 2025

  same root cause · shared Database

• IL-0582 Jobs piled up in Kanmu's PostgreSQL-based qg job queue on Aurora as dequeue slowed and dead tuples kept growingKanmu · 2024

  same root cause · shared Database

• IL-0556 A transaction left open during database maintenance stopped uploads and then downloads on Storj's US1 satelliteStorj · 2024

  same root cause · shared Database

• IL-0531 Azure Database for MySQL Single Server and MariaDB in West US rejected every new connection for a batch of serversMicrosoft Azure · 2024

  same root cause · shared Database

• IL-0565 Cookie Run: Kingdom's CockroachDB cluster failed when 16 of 24 nodes lost their filesystems, causing a 36-hour outageDevsisters · 2022

  same root cause · shared Database

• IL-0207 A routine task run from an engineer's local environment switched to production and deleted GitLab.com's Consul release, so Patroni lost Consul and GitLab.com returned 5xx errors to all trafficGitLab · 2022

  same root cause · shared Database

• IL-0332 A test run against the production database truncated every table and travis-ci.com was down for about 5.5 hoursTravis CI · 2018

  same root cause · shared Database

• IL-0431 During Mackerel's move to AWS, PostgreSQL failed over to a replica that had stopped replicating, losing an hour of dataHatena · 2017

  same root cause · shared Database

• IL-0506 UC Berkeley DataHub logins failed and students got empty home directories after the JupyterHub database was overwrittenUC Berkeley · 2017

  same root cause · shared Database

• IL-0218 GitLab.com's primary PostgreSQL data was wiped by mistake and, with no working backup, the site was down about 18 hoursGitLab · 2017

  same root cause · shared Database

• IL-0573 A developer's mistake dropped nine main tables of the Baemin billing MariaDB database and payments briefly stoppedWoowa Brothers · 2016

  same root cause · shared Database

• IL-0319 Dropping databases on the MySQL replica mysql2 also dropped them on mysql1, as the pair actually ran master-masterOregon State University Open Source Lab · 2015

  same root cause · shared Database

• IL-0337 EBS volumes in one US East Availability Zone got stuck after a network change cut EBS nodes off from their replicasAmazon Web Services · 2011

  same root cause · shared Database

Check your cluster for this

For AI tools and scripts

• This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0576
• The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0576/similar
• MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.