Incident Library
Applied filters ComponentDatabaseFactorNo safeguard on destructive changeDomainKubernetesClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0207 A routine task run from an engineer's local environment switched to production and deleted GitLab.com's Consul release, so Patroni lost Consul and GitLab.com returned 5xx errors to all traffic
    GitLab2022
    Symptoms

    GitLab's monitoring and alerting fired at once. One of the alerts reported a Patroni failover, and users of GitLab.com received 5XX errors; the CDN counted about 13 million requests answered with 503 during the outage.

    Service outageHigh error rateResources deleted
    Trigger

    An engineer ran a routine task with GitLab's tools in a local development environment; during a local deploy or rollback operation the tool's context changed from local development to production, and the change was applied to production.

    Root cause
    Operator error
    Component
    Database
    Contributing factor
    No safeguard on destructive change
    Detection

    GitLab's monitoring and alerting fired immediately; one of the alerts reported a Patroni failover.

    Remediation
    • re-applied the Consul release to the production Kubernetes cluster, with manual steps because the tooling did not support the redeploy directly
    Action items
    • Improve the Kubernetes deployment process to reduce the risk of running harmful deployment operations against production from local environments (https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/16875)
    • Integrate Teleport with the Kubernetes clusters, which could give SREs a read-only role by default (https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/16221)
    Lessons

    A tool that can switch its target from a local environment to production during a deploy or rollback turns routine local work into a production change. GitLab plans to improve its Kubernetes deployment process so that harmful deployment operations are less likely to be run against production from local environments, and to integrate Teleport with its clusters, which could give SREs a read-only role by default. Detection and diagnosis were fast because the alerts led straight from Patroni to its dependency, Consul. Bringing the deleted release back needed manual work, because the tooling did not make the redeploy straightforward, but it was still done relatively quickly.

    View case
  2. IL-0506 UC Berkeley DataHub logins failed and students got empty home directories after the JupyterHub database was overwritten
    UC Berkeley2017
    Symptoms

    At about 1 AM on 9 February 2017, UC Berkeley's DataHub was reported down: users trying to log in got a proxy error. The hub pod was running, but its log was full of sqlite errors.

    Service outageData lossMisrouted requests
    Trigger

    While debugging, an operator ran `.save jupyterhub.sqlite` in sqlite3 inside the hub pod without having attached the database, which replaced the JupyterHub database file with an empty one.

    Root cause
    Operator error
    Component
    DatabasePersistent volumes and stateful setsJupyterHub
    Contributing factor
    No safeguard on destructive changeUnsafe failure behaviourHarmful manual action
    Detection

    DataHub was reported down at about 1 AM, when users got a proxy error at login. At 1:12 AM a user reported that he could log in but all his notebooks were gone.

    Remediation
    • Scaled the hub to 0 replicas
    • Rebuilt the JupyterHub sqlite database by matching Kubernetes PVC names back to user ids
    • Set cookie_id to null instead of a dummy value and attached the rebuilt database to the hub
    • Changed the cookie secret in the hub pod and restarted it to log all users out
    • Deleted all user pods to log everyone out
    • Ran a script that filled in cookie_id for all users, restarted the hub and deleted all user pods again
    • Dropped the new users with a '.' in their name and renamed the older users from '-' to '.'
    Action items
    • Report upstream that JupyterHub uses only part of the cookie secret and discards the rest
    • Report upstream that the cookie_id column of the users table should have UNIQUE and NOT NULL constraints
    • Ask KubeSpawner to support username hashes in PVC and pod names so the names stay the same when the database is deleted
    • Ask OAuthenticator to support using the id from the Google authenticator as the user id, giving stable user ids
    • Switch to Google Cloud SQL, a hosted and managed MySQL database
    • Take regular and tested backups of the database
    • Start an operational FAQ of things to do and not to do
    • Set up better monitoring and paging
    • Document escalation procedures explicitly
    View case
Check your cluster for this

These 2 cases involve Database and JupyterHub. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Operator error2 cases

Component
Database2JupyterHub1Persistent volumes and stateful sets1
Contributing factor
No safeguard on destructive change2Harmful manual action1Unsafe failure behaviour1
Organisation
GitLab1UC Berkeley1
  1. IL-0207 A routine task run from an engineer's local environment switched to production and deleted GitLab.com's Consul release, so Patroni lost Consul and GitLab.com returned 5xx errors to all traffic
    GitLab2022
  2. IL-0506 UC Berkeley DataHub logins failed and students got empty home directories after the JupyterHub database was overwritten
    UC Berkeley2017