Incident Library
Applied filters ComponentIngress controllerSymptomResources deletedFactorNo runbookClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0095 An engineer deleted the prod namespace of the LEAP JupyterHub while believing the kubeconfig pointed at a new cluster, and the hub was down for about an hour
    2i2c2025
    Symptoms

    At 5:00 PM BST on 12 August 2025 a 2i2c engineer declared an incident. While deploying a new cluster called temple and running into NFS issues, the engineer had deleted the prod namespace; the deletion hung, and listing the pods showed many users from a different cluster. The engineer's first message suggested that the kubeconfig had somehow been mutated or corrupted and that some other cluster was being torn down.

    Resources deletedService outage
    Trigger

    While working around a bug in the NFS deployment process for a new hub, an engineer ran a delete of the prod namespace with the kubeconfig context pointing at the LEAP cluster.

    Root cause
    Operator error
    Component
    Persistent volumes and stateful setsJobs and CronJobsIngress controllerTerraform
    Contributing factor
    No safeguard on destructive changeNo runbook
    Detection

    The engineer who ran the deletion noticed when it hung and the pod list showed users of another cluster, and declared the incident at 5:00 PM BST, before the uptime check on leap.2i2c.cloud failed at 5:32 PM.

    Remediation
    • deleted the single remaining user pod in prod so the namespace deletion completed
    • killed the nfs-share-creator job that was blocking the deployment
    • redeployed the prod hub with the updated NFS server IP address
    • deleted the prod ingress during the deployment so that users could not start servers mid-deployment
    Action items
    • Write an incident report
    • Write a short incident response document for the case "I deleted prod, what next?"
    • Upgrade jupyterhub-home-nfs and remove the nfs-share-creator job
    • Add opt-in hardening against nested kubeconfig contexts to the deployer system
    • Change the engineer's shell configuration to show distinct kubeconfig contexts more clearly
    Lessons

    Provisioning persistent storage outside Kubernetes with Terraform meant a deleted namespace cost an hour of downtime and no data. The engineer had no visible cue for which kubeconfig context was active, which turned a workaround on a new cluster into a deletion on a production one. The engineer knew the fix but did not feel confident redeploying without first confirming with the tech lead, in another time zone, that there was no risk of losing persistent data, and reaching the tech lead took time.

    View case
  2. IL-0086 A deployment meant for a CircleCI test Kubernetes cluster also removed Traefik and Calico resources from the production cluster, taking down the CircleCI UI and runner.circleci.com
    CircleCI2021
    Symptoms

    At 15:35 EST on 11 March 2021 the CircleCI web UI and runner.circleci.com became unavailable. The API, which uses a separate ingress point, stayed reachable, and builds kept running.

    Service outageResources deleted
    Trigger

    An update deployed to a test Kubernetes cluster also removed Traefik and Calico resources from CircleCI's primary production cluster.

    Root cause
    Faulty deployment
    Component
    Ingress controllerCalico
    Contributing factor
    Excessive blast radiusNo runbookMonitoring blind spot
    Detection

    CircleCI engineers were alerted and began investigating. The status page posted its first notice at 20:45 UTC.

    Remediation
    • restored Traefik in the production cluster
    • restored Calico in the production cluster, after which pods served customer requests again
    Action items
    • Improve network-level documentation and runbooks to cover every layer of the Kubernetes networks
    • Improve monitoring and alerting for the network infrastructure
    • Add guardrails that isolate the production and staging clusters so they cannot interfere with each other
    Lessons

    A deployment path that can reach both a test cluster and production can remove shared networking components from production. When the ingress controller and the CNI are both affected, restoring the ingress alone is not enough: engineers had to find the link between Calico and Traefik and restore Calico before pods served traffic again. Time went to DNS and AWS red herrings because nothing linked the outage to the test deployment; CircleCI's answer is better network runbooks, network alerting and guardrails between clusters.

    View case
Check your cluster for this

These 2 cases involve Ingress controller and Calico. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause2

Operator error1 case

Component
Ingress controllerJobs and CronJobsPersistent volumes and stateful setsTerraform
Contributing factor
No runbookNo safeguard on destructive change
Organisation
2i2c
  1. IL-0095 An engineer deleted the prod namespace of the LEAP JupyterHub while believing the kubeconfig pointed at a new cluster, and the hub was down for about an hour
    2i2c2025

Faulty deployment1 case

Component
CalicoIngress controller
Contributing factor
Excessive blast radiusMonitoring blind spotNo runbook
Organisation
CircleCI
  1. IL-0086 A deployment meant for a CircleCI test Kubernetes cluster also removed Traefik and Calico resources from the production cluster, taking down the CircleCI UI and runner.circleci.com
    CircleCI2021