Cases4 cases

1 to 4 of 4

1. IL-0095 An engineer deleted the prod namespace of the LEAP JupyterHub while believing the kubeconfig pointed at a new cluster, and the hub was down for about an hour

   2i2c2025

   Symptoms

   At 5:00 PM BST on 12 August 2025 a 2i2c engineer declared an incident. While deploying a new cluster called temple and running into NFS issues, the engineer had deleted the prod namespace; the deletion hung, and listing the pods showed many users from a different cluster. The engineer's first message suggested that the kubeconfig had somehow been mutated or corrupted and that some other cluster was being torn down.

   Resources deletedService outage

   Trigger

   While working around a bug in the NFS deployment process for a new hub, an engineer ran a delete of the prod namespace with the kubeconfig context pointing at the LEAP cluster.

   Root cause

   Operator error

   Component

   Persistent volumes and stateful setsJobs and CronJobsIngress controllerTerraform

   Contributing factor

   No safeguard on destructive changeNo runbook

   Detection

   The engineer who ran the deletion noticed when it hung and the pod list showed users of another cluster, and declared the incident at 5:00 PM BST, before the uptime check on leap.2i2c.cloud failed at 5:32 PM.

   Remediation

   • deleted the single remaining user pod in prod so the namespace deletion completed
   • killed the nfs-share-creator job that was blocking the deployment
   • redeployed the prod hub with the updated NFS server IP address
   • deleted the prod ingress during the deployment so that users could not start servers mid-deployment

   Action items

   • Write an incident report
   • Write a short incident response document for the case "I deleted prod, what next?"
   • Upgrade jupyterhub-home-nfs and remove the nfs-share-creator job
   • Add opt-in hardening against nested kubeconfig contexts to the deployer system
   • Change the engineer's shell configuration to show distinct kubeconfig contexts more clearly

   Lessons

   Provisioning persistent storage outside Kubernetes with Terraform meant a deleted namespace cost an hour of downtime and no data. The engineer had no visible cue for which kubeconfig context was active, which turned a workaround on a new cluster into a deletion on a production one. The engineer knew the fix but did not feel confident redeploying without first confirming with the tech lead, in another time zone, that there was no risk of losing persistent data, and reaching the tech lead took time.

   View case
2. IL-0522 An ingress-nginx chart upgrade deleted the jenkins.io Ingress and Fastly served HTTP/503 errors for hours

   Jenkins project2021

   Symptoms

   At 11:32 UTC on 18 November 2021, a user reported in the jenkinsci/jenkins Gitter channel that https://www.jenkins.io/doc/pipeline/steps/ was answering HTTP/503 errors with the message "Guru meditation" from Fastly's varnish.

   Service outageResources deleted

   Trigger

   The ingress-nginx Helm chart was upgraded from 4.0.6 to 4.0.8 at 10:42 UTC, and the new chart added an admission webhook rule that forbids Nginx configuration blocks in Ingress annotations.

   Root cause

   Breaking change in an upgrade

   Component

   Ingress controllerAdmission webhookHelmCDN and edge network

   Contributing factor

   Unverified assumptionCache-masked failureMonitoring blind spotHarmful manual action

   Detection

   A user reported in the jenkinsci/jenkins Gitter channel at 11:32 UTC that a jenkins.io page answered HTTP/503 errors, and status.jenkins.io reported the incident at 11:49 UTC.

   Remediation

   • Disabled the automated systems, rolled the ingress-nginx chart back to 4.0.6 and deleted the admission webhook by hand
   • Removed the forbidden annotations from the jenkins.io configuration
   • Upgraded the jenkins.io Helm chart to the last version known to work with ingress-nginx 4.0.8
   • Redeployed the jenkins.io Helm chart through the automated system and told Fastly to drop all cached pages

   Action items

   • Always read the changelog in depth when updating the ingress controllers, with two people pairing in real time
   • Add custom Datadog monitoring for www.origin.jenkins.io so that such an outage is detected
   • Make the monitor send the Host header www.jenkins.io so it is not redirected to Fastly
   • Check a set of pages, such as the pipeline steps page, not only the front page
   • Consider an ephemeral k3s end-to-end test before deploying to production

   Lessons

   Trusting a patch version of a Helm chart for the Ingress controller without reading its changelog in depth let a new admission rule remove the jenkins.io Ingress in production. Without monitoring on origin.jenkins.io, a user found the outage. A complete staging cluster with the same DNS setup would have caught the problem before production.

   View case
3. IL-0086 A deployment meant for a CircleCI test Kubernetes cluster also removed Traefik and Calico resources from the production cluster, taking down the CircleCI UI and runner.circleci.com

   CircleCI2021

   Symptoms

   At 15:35 EST on 11 March 2021 the CircleCI web UI and runner.circleci.com became unavailable. The API, which uses a separate ingress point, stayed reachable, and builds kept running.

   Service outageResources deleted

   Trigger

   An update deployed to a test Kubernetes cluster also removed Traefik and Calico resources from CircleCI's primary production cluster.

   Root cause

   Faulty deployment

   Component

   Ingress controllerCalico

   Contributing factor

   Excessive blast radiusNo runbookMonitoring blind spot

   Detection

   CircleCI engineers were alerted and began investigating. The status page posted its first notice at 20:45 UTC.

   Remediation

   • restored Traefik in the production cluster
   • restored Calico in the production cluster, after which pods served customer requests again

   Action items

   • Improve network-level documentation and runbooks to cover every layer of the Kubernetes networks
   • Improve monitoring and alerting for the network infrastructure
   • Add guardrails that isolate the production and staging clusters so they cannot interfere with each other

   Lessons

   A deployment path that can reach both a test cluster and production can remove shared networking components from production. When the ingress controller and the CNI are both affected, restoring the ingress alone is not enough: engineers had to find the link between Calico and Traefik and restore Calico before pods served traffic again. Time went to DNS and AWS red herrings because nothing linked the outage to the test deployment; CircleCI's answer is better network runbooks, network alerting and guardrails between clusters.

   View case
4. IL-0089 A test cluster deletion script run with the kube context pointing at the live-1 production cluster deleted its ingress controller, monitoring, logging, cert-manager, kiam and external-dns

   UK Ministry of Justice2020

   Symptoms

   At 18:27 on 21 September 2020 the Cloud Platform team detected that some services running on the live-1 production cluster had gone down, and some users could not reach them. The incident was declared at 18:40.

   Service outageResources deletedMonitoring outageStale endpoint routing

   Trigger

   A script for deleting a test cluster was triggered while the kube context was pointing at the live-1 production cluster instead of the test cluster.

   Root cause

   Operator error

   Component

   Ingress controllerCredential broker for podsexternal-dnsCloud load balancerPrometheus

   Contributing factor

   No safeguard on destructive change

   Detection

   The first detection was at 18:27, and the incident was declared 13 minutes later, at 18:40.

   Remediation

   • restored the default ingress controller together with external-dns and kiam, which it depends on
   • corrected the owner field in the Formbuilder Route 53 TXT records so external-dns pointed their A records at the new NLB
   • restored the remaining components with a targeted apply
   • ran the apply pipeline to restore all certificates, ServiceMonitors and Prometheus rules from the environment repository

   View case

By root cause3