Incident Library
Applied filters Root causeOperator errorSymptomData lossOrganisationCareNetClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0576 Every dataset in CareNet's production BigQuery data warehouse was deleted, and UNDROP SCHEMA could not restore them
    CareNet2025
    Symptoms

    On a day in August 2025, at 10:41, CareNet's system development division posted a first report on Slack that a failure was occurring in BigQuery, the production data warehouse. At 11:47 the division's investigation found that the production datasets had been deleted by a service account belonging to the AI technology development team.

    Data loss
    Trigger

    Test code that was being modified contained a step that resets test data by deleting a dataset and recreating it with the same name. Because of a configuration mistake, the step ran against production and deleted and recreated every production dataset.

    Root cause
    Operator error
    Component
    Database
    Contributing factor
    No safeguard on destructive change
    Detection

    The system development division posted the first report of a BigQuery failure on Slack at 10:41, and its investigation found at 11:47 that the AI technology development team's service account had deleted the production datasets.

    Remediation
    • Restored the tables one at a time with bq cp in Cloud Shell from BigQuery time travel, to a point before the incident
    • Recreated views and table functions, which bq cp cannot restore, from the creation queries kept in BigQuery saved queries
    Action items
    • Enforce controls so that test code cannot access the production environment
    • Remove the excessive permissions granted to the service account
    • Keep dataset and table information as managed metadata, because restoring with bq cp requires knowing the table names
    • Keep the creation queries of views and table functions in a place such as GitHub, because bq cp cannot restore them
    • Consider how BigQuery should be backed up
    Lessons

    BigQuery time travel with bq cp recovered most of the data, but only because the table names were known, and it does not bring back views or table functions. Test code should not be able to reach production, and service accounts should not hold more permissions than they need.

    View case
Check your cluster for this

This case involves Database. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Operator error1 case

Component
Database
Contributing factor
No safeguard on destructive change
Organisation
CareNet
  1. IL-0576 Every dataset in CareNet's production BigQuery data warehouse was deleted, and UNDROP SCHEMA could not restore them
    CareNet2025