Cases2 cases

1 to 2 of 2

1. IL-0095 An engineer deleted the prod namespace of the LEAP JupyterHub while believing the kubeconfig pointed at a new cluster, and the hub was down for about an hour

   2i2c2025

   Symptoms

   At 5:00 PM BST on 12 August 2025 a 2i2c engineer declared an incident. While deploying a new cluster called temple and running into NFS issues, the engineer had deleted the prod namespace; the deletion hung, and listing the pods showed many users from a different cluster. The engineer's first message suggested that the kubeconfig had somehow been mutated or corrupted and that some other cluster was being torn down.

   Resources deletedService outage

   Trigger

   While working around a bug in the NFS deployment process for a new hub, an engineer ran a delete of the prod namespace with the kubeconfig context pointing at the LEAP cluster.

   Root cause

   Operator error

   Component

   Persistent volumes and stateful setsJobs and CronJobsIngress controllerTerraform

   Contributing factor

   No safeguard on destructive changeNo runbook

   Detection

   The engineer who ran the deletion noticed when it hung and the pod list showed users of another cluster, and declared the incident at 5:00 PM BST, before the uptime check on leap.2i2c.cloud failed at 5:32 PM.

   Remediation

   • deleted the single remaining user pod in prod so the namespace deletion completed
   • killed the nfs-share-creator job that was blocking the deployment
   • redeployed the prod hub with the updated NFS server IP address
   • deleted the prod ingress during the deployment so that users could not start servers mid-deployment

   Action items

   • Write an incident report
   • Write a short incident response document for the case "I deleted prod, what next?"
   • Upgrade jupyterhub-home-nfs and remove the nfs-share-creator job
   • Add opt-in hardening against nested kubeconfig contexts to the deployer system
   • Change the engineer's shell configuration to show distinct kubeconfig contexts more clearly

   Lessons

   Provisioning persistent storage outside Kubernetes with Terraform meant a deleted namespace cost an hour of downtime and no data. The engineer had no visible cue for which kubeconfig context was active, which turned a workaround on a new cluster into a deletion on a production one. The engineer knew the fix but did not feel confident redeploying without first confirming with the tech lead, in another time zone, that there was no risk of losing persistent data, and reaching the tech lead took time.

   View case
2. IL-0197 Moving a three-node Elasticsearch StatefulSet to new persistent volumes split the cluster in two, because the Helm chart hardcoded cluster.initial_master_nodes to elasticsearch-0

   Palark2021

   Symptoms

   After the last old Elasticsearch pod, elasticsearch-0, was deleted and came back on a new, empty persistent volume, `_cat/nodes` run inside that pod listed a single node: elasticsearch-0 itself, marked as master. The same command run inside elasticsearch-1 listed the other five nodes, elasticsearch-1 to elasticsearch-5, with elasticsearch-2 as master.

   Split brain

   Trigger

   Deleting the elasticsearch-0 pod and its PVC so that it restarted on an empty volume of the new StorageClass.

   Root cause

   Operator error

   Component

   Elasticsearch or OpenSearchPersistent volumes and stateful setsHelm

   Contributing factor

   Unverified assumption

   Detection

   The engineer running the migration checked `_cat/nodes` after elasticsearch-0 restarted and saw it alone in a cluster of its own.

   Remediation

   • scaled the three deployments of workers writing to Elasticsearch down to zero
   • listed elasticsearch-0, elasticsearch-1 and elasticsearch-2 under initial_master_nodes in the elasticsearch ConfigMap and removed the cluster.initial_master_nodes environment variable from the StatefulSet
   • saved and deleted the elasticsearch discovery Service and deleted the PVC of elasticsearch-0 while the rolling update ran
   • set initial_master_nodes to elasticsearch-3, elasticsearch-4 and elasticsearch-5, created an empty PVC for elasticsearch-0 and deleted all six pods
   • recreated the discovery Service from the saved manifest, after which the six nodes formed one cluster with every index green

   Action items

   • Restore the original initial_master_nodes values in the ConfigMap and restart all pods again
   • Move all shards to nodes 0, 1 and 2 and scale the cluster down from six nodes to three
   • Commit all the manual changes to the repository

   Lessons

   A setting that only matters when a cluster first starts, such as Elasticsearch's cluster.initial_master_nodes, can sit in a Helm chart template rather than in the ConfigMap that was reviewed. When a pod named in it starts with an empty data directory, it can bootstrap a new cluster beside the existing one. The author's conclusion is that a migration plan like this one breaks on any surprise and needs a plan B ready. Before running it in a more important environment, rehearse it in a test environment configured like production, and either schedule downtime or move the load to another cluster: stop the writers, take a fresh backup, then move the data.

   View case

By root cause1