Incident Library
Applied filters Root causeOperator errorSymptomService outageFactorNo safeguard on destructive changeClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases4 cases

1 to 4 of 4

  1. IL-0095 An engineer deleted the prod namespace of the LEAP JupyterHub while believing the kubeconfig pointed at a new cluster, and the hub was down for about an hour
    2i2c2025
    Symptoms

    At 5:00 PM BST on 12 August 2025 a 2i2c engineer declared an incident. While deploying a new cluster called temple and running into NFS issues, the engineer had deleted the prod namespace; the deletion hung, and listing the pods showed many users from a different cluster. The engineer's first message suggested that the kubeconfig had somehow been mutated or corrupted and that some other cluster was being torn down.

    Resources deletedService outage
    Trigger

    While working around a bug in the NFS deployment process for a new hub, an engineer ran a delete of the prod namespace with the kubeconfig context pointing at the LEAP cluster.

    Root cause
    Operator error
    Component
    Persistent volumes and stateful setsJobs and CronJobsIngress controllerTerraform
    Contributing factor
    No safeguard on destructive changeNo runbook
    Detection

    The engineer who ran the deletion noticed when it hung and the pod list showed users of another cluster, and declared the incident at 5:00 PM BST, before the uptime check on leap.2i2c.cloud failed at 5:32 PM.

    Remediation
    • deleted the single remaining user pod in prod so the namespace deletion completed
    • killed the nfs-share-creator job that was blocking the deployment
    • redeployed the prod hub with the updated NFS server IP address
    • deleted the prod ingress during the deployment so that users could not start servers mid-deployment
    Action items
    • Write an incident report
    • Write a short incident response document for the case "I deleted prod, what next?"
    • Upgrade jupyterhub-home-nfs and remove the nfs-share-creator job
    • Add opt-in hardening against nested kubeconfig contexts to the deployer system
    • Change the engineer's shell configuration to show distinct kubeconfig contexts more clearly
    Lessons

    Provisioning persistent storage outside Kubernetes with Terraform meant a deleted namespace cost an hour of downtime and no data. The engineer had no visible cue for which kubeconfig context was active, which turned a workaround on a new cluster into a deletion on a production one. The engineer knew the fix but did not feel confident redeploying without first confirming with the tech lead, in another time zone, that there was no risk of losing persistent data, and reaching the tech lead took time.

    View case
  2. IL-0207 A routine task run from an engineer's local environment switched to production and deleted GitLab.com's Consul release, so Patroni lost Consul and GitLab.com returned 5xx errors to all traffic
    GitLab2022
    Symptoms

    GitLab's monitoring and alerting fired at once. One of the alerts reported a Patroni failover, and users of GitLab.com received 5XX errors; the CDN counted about 13 million requests answered with 503 during the outage.

    Service outageHigh error rateResources deleted
    Trigger

    An engineer ran a routine task with GitLab's tools in a local development environment; during a local deploy or rollback operation the tool's context changed from local development to production, and the change was applied to production.

    Root cause
    Operator error
    Component
    Application database
    Contributing factor
    No safeguard on destructive change
    Detection

    GitLab's monitoring and alerting fired immediately; one of the alerts reported a Patroni failover.

    Remediation
    • re-applied the Consul release to the production Kubernetes cluster, with manual steps because the tooling did not support the redeploy directly
    Action items
    • Improve the Kubernetes deployment process to reduce the risk of running harmful deployment operations against production from local environments (https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/16875)
    • Integrate Teleport with the Kubernetes clusters, which could give SREs a read-only role by default (https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/16221)
    Lessons

    A tool that can switch its target from a local environment to production during a deploy or rollback turns routine local work into a production change. GitLab plans to improve its Kubernetes deployment process so that harmful deployment operations are less likely to be run against production from local environments, and to integrate Teleport with its clusters, which could give SREs a read-only role by default. Detection and diagnosis were fast because the alerts led straight from Patroni to its dependency, Consul. Bringing the deleted release back needed manual work, because the tooling did not make the redeploy straightforward, but it was still done relatively quickly.

    View case
  3. IL-0115 One of OpenAI's three production AKS clusters went offline through a human error, and rebuilding it took 10½ hours because an AKS Kubernetes upgrade broke the ingress and AKS cannot be downgraded
    OpenAI2022
    Symptoms

    At 18:56 PDT on 23 June 2022 (01:56 UTC on 24 June) several models went offline at once: some fine-tuned curie and davinci models, the Codex beta models code-davinci-001 and code-cushman-001, the legacy GPT models curie, babbage and ada, and all Embeddings models.

    Service outageHigh error rateIntermittent timeouts
    Trigger

    A destructive operation on one of the three production clusters, caused by human error that the existing safeguards did not catch, took the whole cluster offline.

    Root cause
    Operator error
    Component
    Ingress controllerModel hub
    Contributing factor
    No safeguard on destructive changeNo headroomDeferred upgradesOverlapping failures
    Remediation
    • re-allocated the models from the lost cluster to the remaining two clusters
    • rebuilt the VPN link with Azure support after losing the connection to AKS during cluster setup
    • built workarounds with Azure support after losing the connection to Azure Private Link
    • tore down the cluster and rebuilt it from scratch after the upgraded Kubernetes version broke the ingress
    • moved traffic back into the rebuilt cluster between 05:30 and 06:30 PDT
    Action items
    • Add checks and locks so that this and similar production-critical infrastructure cannot be modified by accident
    • Add better visibility into the operations being performed and make parts of the system more lexicographically distinct
    • Keep working with Azure on the AKS network connectivity issues so they do not disrupt new cluster operations
    • Update playbooks with the new cluster failure modes and change the setup steps for new clusters and infrastructure
    • Upgrade Kubernetes systematically and set aside time so the clusters do not fall behind again
    • Shard fine-tuned models across multiple clusters and work towards more automatic cross-cluster replication
    • Distribute and replicate Azure Blob Storage so fine-tuned models load faster and more reliably
    • Finish rolling out node-to-node caching to relieve Azure Blob Storage bottlenecks
    • Change how model weights are stored so they are more efficient and faster to load
    Lessons

    Losing one of three clusters exhausted the GPU headroom, and GPUs could not be added on short notice, so full recovery depended on bringing the same cluster back. Rebuilding it exposed a deferred Kubernetes upgrade: the newer version broke the ingress, and because AKS cannot downgrade there was no way back except a full rebuild. Recovering many large models at once is limited by the bandwidth of the storage they load from, and the caches that normally hide this do not help when everything moves to a different cluster.

    View case
  4. IL-0089 A test cluster deletion script run with the kube context pointing at the live-1 production cluster deleted its ingress controller, monitoring, logging, cert-manager, kiam and external-dns
    UK Ministry of Justice2020
    Symptoms

    At 18:27 on 21 September 2020 the Cloud Platform team detected that some services running on the live-1 production cluster had gone down, and some users could not reach them. The incident was declared at 18:40.

    Service outageResources deletedMonitoring outageStale endpoint routing
    Trigger

    A script for deleting a test cluster was triggered while the kube context was pointing at the live-1 production cluster instead of the test cluster.

    Root cause
    Operator error
    Component
    Ingress controllerCredential broker for podsexternal-dnsCloud load balancerPrometheus
    Contributing factor
    No safeguard on destructive change
    Detection

    The first detection was at 18:27, and the incident was declared 13 minutes later, at 18:40.

    Remediation
    • restored the default ingress controller together with external-dns and kiam, which it depends on
    • corrected the owner field in the Formbuilder Route 53 TXT records so external-dns pointed their A records at the new NLB
    • restored the remaining components with a targeted apply
    • ran the apply pipeline to restore all certificates, ServiceMonitors and Prometheus rules from the environment repository
    View case
Check your cluster for this

These 4 cases involve Ingress controller and Cloud load balancer. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Operator error4 cases

Component
Ingress controller3Cloud load balancer1Credential broker for pods1Application database1external-dns1Jobs and CronJobs1Model hub1Persistent volumes and stateful sets1Prometheus1Terraform1
Contributing factor
No safeguard on destructive change4No runbook1No headroom1Overlapping failures1Deferred upgrades1
Organisation
2i2c1GitLab1OpenAI1UK Ministry of Justice1
  1. IL-0095 An engineer deleted the prod namespace of the LEAP JupyterHub while believing the kubeconfig pointed at a new cluster, and the hub was down for about an hour
    2i2c2025
  2. IL-0207 A routine task run from an engineer's local environment switched to production and deleted GitLab.com's Consul release, so Patroni lost Consul and GitLab.com returned 5xx errors to all traffic
    GitLab2022
  3. IL-0115 One of OpenAI's three production AKS clusters went offline through a human error, and rebuilding it took 10½ hours because an AKS Kubernetes upgrade broke the ingress and AKS cannot be downgraded
    OpenAI2022
  4. IL-0089 A test cluster deletion script run with the kube context pointing at the live-1 production cluster deleted its ingress controller, monitoring, logging, cert-manager, kiam and external-dns
    UK Ministry of Justice2020