Incident Library
Applied filters SymptomService outageDomainKubernetesClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases96 cases

54 to 58 of 96

  1. IL-0115 One of OpenAI's three production AKS clusters went offline through a human error, and rebuilding it took 10½ hours because an AKS Kubernetes upgrade broke the ingress and AKS cannot be downgraded
    OpenAI2022
    Symptoms

    At 18:56 PDT on 23 June 2022 (01:56 UTC on 24 June) several models went offline at once: some fine-tuned curie and davinci models, the Codex beta models code-davinci-001 and code-cushman-001, the legacy GPT models curie, babbage and ada, and all Embeddings models.

    Service outageHigh error rateIntermittent timeouts
    Trigger

    A destructive operation on one of the three production clusters, caused by human error that the existing safeguards did not catch, took the whole cluster offline.

    Root cause
    Operator error
    Component
    Ingress controllerModel hub
    Contributing factor
    No safeguard on destructive changeNo headroomDeferred upgradesOverlapping failures
    Remediation
    • re-allocated the models from the lost cluster to the remaining two clusters
    • rebuilt the VPN link with Azure support after losing the connection to AKS during cluster setup
    • built workarounds with Azure support after losing the connection to Azure Private Link
    • tore down the cluster and rebuilt it from scratch after the upgraded Kubernetes version broke the ingress
    • moved traffic back into the rebuilt cluster between 05:30 and 06:30 PDT
    Action items
    • Add checks and locks so that this and similar production-critical infrastructure cannot be modified by accident
    • Add better visibility into the operations being performed and make parts of the system more lexicographically distinct
    • Keep working with Azure on the AKS network connectivity issues so they do not disrupt new cluster operations
    • Update playbooks with the new cluster failure modes and change the setup steps for new clusters and infrastructure
    • Upgrade Kubernetes systematically and set aside time so the clusters do not fall behind again
    • Shard fine-tuned models across multiple clusters and work towards more automatic cross-cluster replication
    • Distribute and replicate Azure Blob Storage so fine-tuned models load faster and more reliably
    • Finish rolling out node-to-node caching to relieve Azure Blob Storage bottlenecks
    • Change how model weights are stored so they are more efficient and faster to load
    Lessons

    Losing one of three clusters exhausted the GPU headroom, and GPUs could not be added on short notice, so full recovery depended on bringing the same cluster back. Rebuilding it exposed a deferred Kubernetes upgrade: the newer version broke the ingress, and because AKS cannot downgrade there was no way back except a full rebuild. Recovering many large models at once is limited by the bandwidth of the storage they load from, and the caches that normally hide this do not help when everything moves to a different cluster.

    View case
  2. IL-0403 mybinder.org could not create pods for hours as the GKE gke-resource-quotas quota wrongly reported 15k pods in use
    Binder Project2022
    Symptoms

    On 2 June 2022 from 10:00 CET, mybinder.org stopped launching any new pods, and nobody on the team noticed until a user reported a Binder outage in the Matrix channel at 21:00.

    Deploys failingService outage
    Root cause
    Software defect
    Component
    GKE control plane
    Contributing factor
    Monitoring blind spot
    Detection

    A user reported a Binder outage in the Matrix channel at 21:00, and a team member who saw it found that pods had not been launching for several hours.

    Remediation
    • Deleted the gke-resource-quotas ResourceQuota, after which pods could be created again
    Action items
    • Address uptime monitoring and alerting
    Lessons

    It took 11 hours to realise there was a major outage. Once the right object was known, deleting it fixed the problem quickly, and the team was lucky that someone with the skills and permissions to make the change was at their computer at 11pm their time.

    View case
  3. IL-0113 The managed GKE control plane of Webhook Relay's main cluster failed because its API server certificates had never been rotated, and a manual credential rotation reported success without taking effect
    Webhook Relay2022
    Symptoms

    At 12:00 EST the health check system reported that the backend APIs, the frontend and the tunnels in the EU region were down. The team first took it for a transient error, such as an interrupted client request or a server evicted for memory or CPU pressure, but the main dashboard was offline as well.

    Service outageControl plane lagControl plane unresponsiveStale or inconsistent API readsCredentials unavailable
    Root cause
    Certificate or credential expiry
    Component
    GKE control planeKubernetes API serverNode poolRBAC and service accounts
    Contributing factor
    Limited debug accessSingle point of failure
    Detection

    The health check system sent a notification that the service was down; the team then found the main dashboard offline.

    Remediation
    • recreated the node pools by hand after the credential rotation failed to recreate them
    • duplicated the main cluster's services into a newly created GKE cluster
    • deleted the Kubernetes Service records in the old cluster so its load balancers could be attached to the new cluster
    • detached the StatefulSet from the old cluster and switched all writes to the main disk
    • exported the data from the temporary ClickHouse instance and inserted it into the main instance
    Action items
    • Change the deployment manifests so services can be switched between clusters quickly without a complicated persistent disk migration
    Lessons

    Recovery of existing infrastructure needs a time limit. Throughout the outage the old cluster looked about to recover, and the reluctance to give up on the salvage effort kept the team from starting over with a new cluster much sooner, which would have been the right call. On older GKE clusters, check the validity of the cluster CA certificate, for example by decoding masterAuth.clusterCaCertificate from gcloud container clusters describe and reading it with openssl x509. GKE's multiple availability zones made no difference in this failure.

    View case
  4. IL-0517 Gitpod workspaces failed to start worldwide and lost data when CoreDNS lookups to 8.8.8.8 timed out in its clusters
    Gitpod2022
    Symptoms

    At around 5.44pm UTC on 25 January 2022, Gitpod saw a high error rate when pulling images from Google Container Registry (gcr.io), with the error "cannot get manifest". About 10 minutes later users reported that they could not start workspaces.

    Image pull failuresDNS resolution failuresService outageData loss
    Root cause
    DNS or network fault
    Component
    CoreDNSContainer registry and its credentialsObject storage
    Contributing factor
    Single point of failure
    Detection

    The team noticed a high error rate on image pulls from GCR at around 5.44pm UTC; about 10 minutes later users reported that workspaces would not start, and the on-call team started incident response.

    Remediation
    • Created new workspace clusters in the EU and the US with a werft job
    • Redirected traffic to the new clusters
    Action items
    • Keep workspace data on cloud provider persistent volumes that survive node and cluster failures
    • Let users download their files from a workspace that cannot be started
    • Improve the workspace backup process
    • Use three DNS servers instead of one: a GCP-internal one, 8.8.8.8 and Cloudflare's 1.1.1.1
    • Do more DNS resiliency work, tracked in an epic
    • Improve support for multi-region clusters and later go multi-cloud
    View case
  5. IL-0522 An ingress-nginx chart upgrade deleted the jenkins.io Ingress and Fastly served HTTP/503 errors for hours
    Jenkins project2021
    Symptoms

    At 11:32 UTC on 18 November 2021, a user reported in the jenkinsci/jenkins Gitter channel that https://www.jenkins.io/doc/pipeline/steps/ was answering HTTP/503 errors with the message "Guru meditation" from Fastly's varnish.

    Service outageResources deleted
    Trigger

    The ingress-nginx Helm chart was upgraded from 4.0.6 to 4.0.8 at 10:42 UTC, and the new chart added an admission webhook rule that forbids Nginx configuration blocks in Ingress annotations.

    Root cause
    Breaking change in an upgrade
    Component
    Ingress controllerAdmission webhookHelmCDN and edge network
    Contributing factor
    Unverified assumptionCache-masked failureMonitoring blind spotHarmful manual action
    Detection

    A user reported in the jenkinsci/jenkins Gitter channel at 11:32 UTC that a jenkins.io page answered HTTP/503 errors, and status.jenkins.io reported the incident at 11:49 UTC.

    Remediation
    • Disabled the automated systems, rolled the ingress-nginx chart back to 4.0.6 and deleted the admission webhook by hand
    • Removed the forbidden annotations from the jenkins.io configuration
    • Upgraded the jenkins.io Helm chart to the last version known to work with ingress-nginx 4.0.8
    • Redeployed the jenkins.io Helm chart through the automated system and told Fastly to drop all cached pages
    Action items
    • Always read the changelog in depth when updating the ingress controllers, with two people pairing in real time
    • Add custom Datadog monitoring for www.origin.jenkins.io so that such an outage is detected
    • Make the monitor send the Host header www.jenkins.io so it is not redirected to Fastly
    • Check a set of pages, such as the pipeline steps page, not only the front page
    • Consider an ephemeral k3s end-to-end test before deploying to production
    Lessons

    Trusting a patch version of a Helm chart for the Ingress controller without reading its changelog in depth let a new admission rule remove the jenkins.io Ingress in production. Without monitoring on origin.jenkins.io, a user found the outage. A complete staging cluster with the same DNS setup would have caught the problem before production.

    View case
Check your cluster for this

These 96 cases involve Node pool and Ingress controller. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause11

Capacity exhaustion17 cases

Component
Database4Container memory limit4Node pool4CoreDNS3Persistent volumes and stateful sets3In-memory cache2Helm2Ingress controller2kubelet2Linux kernel2Pod scheduling and placement2Admission webhook1BGP routing1Cloud block storage1Cloud controller manager1CNI plugin1conntrack table1Container runtime1Client-side DNS resolver and cache1Horizontal pod autoscaler1InfiniBand fabric1Istio1Jobs and CronJobs1JupyterHub1kube-dns1Model hub1NAT gateway1Network control plane (SDN)1NIC, cable and switch port link1Node root disk1Container registry and its credentials1
Contributing factor
Single point of failure7Automation amplification6Monitoring blind spot5Unverified assumption4Excessive blast radius4Unreviewed defaults4Retry storm3Change during peak hours2Unknown limit2Harmful manual action2No caching2No resource requests or limits2Ignored early signal1Limited debug access1No alert on limit1No capacity planning1No runbook1No headroom1Unrepresentative staging1
Organisation
Backpackr22i2c1BC Government1Binder Project1Buildkite1Datadog1freedesktop.org1Honeycomb1IONOS Cloud1Magic Pages1Mercari1Moonlight1RebelMouse1Saltside1Tinder1Universe1
  1. IL-0102 CoreDNS pods on a production EKS cluster ran out of memory under a burst of new pods, endpoints and nodes, taking down the Buildkite web UI, APIs and job dispatch
    Buildkite2026
  2. IL-0350 Hosts in an IONOS TXL cluster lost connectivity when an InfiniBand fabric resweep failed on a full multicast table
    IONOS Cloud2026
  3. IL-0067 Redis filled its persistent volume, corrupted its append-only file and crash-looped on every restart
    BC Government2025
  4. IL-0080 kube-dns on Mercari's GKE clusters could not keep up with peak-hour DNS query rates, and a Sev1 production outage followed when several services failed DNS resolution
    Mercari2025
  5. IL-0082 Losing one CoreDNS pod during a planned EKS node reroll overloaded the remaining CoreDNS pods, causing DNS timeouts and errors across multiple services at RebelMouse
    RebelMouse2025
  6. IL-0170 During a Lunar New Year event, traffic more than three times normal filled the nf_conntrack table on the EKS node running the Istio ingress gateway, dropping packets and taking down Backpackr's web and app services
    Backpackr2025
  7. IL-0027 All CoreDNS pods landed on one node, the node ran out of memory, and DNS for the whole service went down
    Backpackr2025
  8. IL-0085 A control plane node of Magic Pages' Kubernetes cluster on Hetzner Cloud ran out of memory, the cluster's own calls then used up the Hetzner API rate limit, and all customer websites were down for about 7.5 hours
    Magic Pages2025
  9. IL-0202 A burst of queries and database writes exhausted connection pools and memory on Honeycomb's ingest hosts, and Kubernetes crash-loop back-off kept failed hosts offline so ingest capacity stayed too low to recover
    Honeycomb2023
  10. IL-0094 Student pods filled the shared nodes of a JupyterHub cluster and starved the hub's core pods, freezing the hub
    2i2c2020
  11. IL-0037 Image registry rate-limited a whole region after a crash-looping DaemonSet with imagePullPolicy Always pulled through three NAT instances
    Datadog2019
  12. IL-0064 CPU-heavy pods packed onto one Google Cloud node drove it to kernel panics, and the crash moved from node to node until the site was down
    Moonlight2019
  13. IL-0472 Node-to-pod traffic and DNS failed on Tinder's 605-node Flannel cluster when every node's ARP cache passed gc_thresh3
    Tinder2019
  14. IL-0518 gitlab.freedesktop.org on GKE ran short of volume space and a planned 30-minute storage upgrade became a 4-hour outage
    freedesktop.org2018
  15. IL-0503 mybinder.org launches failed after a GKE node ran out of memory and its nginx-ingress proxy pod crash-looped
    Binder Project2017
  16. IL-0036 A reindex Job that kept failing exhausted one node after another until the whole cluster was down
    Universe2017
  17. IL-0040 Parallel Helm installs saturated small gp2 root volumes on a kops cluster and left all three workers NotReady with the kubelet or Docker down
    Saltside2017

Dependency failure3 cases

Component
Cloud block storage1Database1Data center power and cooling1etcd1Liveness and readiness probes1InfiniBand fabric1Persistent volumes and stateful sets1
Contributing factor
Monitoring blind spot2Single point of failure2Automation amplification1Excessive blast radius1Unreviewed defaults1Harmful manual action1No alert on limit1No runbook1Overlapping failures1
Organisation
Magic Pages1Nebius1Yonghyun Kim1
  1. IL-0133 A storm shut down cooling at Nebius's us-central1 data center, and a disk hot-plug bug kept Managed Kubernetes control planes down for hours after power returned
    Nebius2026
  2. IL-0187 A brief Spring Cloud Config Server downtime failed every client application's liveness probe, because the actuator health check included the config server, and Kubernetes restarted the pods in a kill loop
    Yonghyun Kim2025
  3. IL-0489 Rejoining a failed node of Magic Pages' three-node MySQL cluster broke its metadata and took customer websites down
    Magic Pages2025

Cloud provider outage9 cases

Component
Database3AKS control plane2Cluster autoscaler2Ingress controller2Kubernetes API server2Node pool2CDN and edge network1Cloud load balancer1Coordination and key-value store1GKE control plane1Helm1Horizontal pod autoscaler1kubelet1Node root disk1Persistent volumes and stateful sets1Secrets store1Web application firewall rules1
Contributing factor
Overlapping failures3Single point of failure3Unverified assumption2Excessive blast radius2Automation amplification1Ignored early signal1Unpatched known bug1Limited debug access1No capacity planning1No headroom1Pages not delivered1Unrepresentative staging1
Organisation
2i2c1ANDPAD1Devsisters1Grafana Labs1loveholidays1Microsoft Azure1Namespace1Python Software Foundation1UC Berkeley1
  1. IL-0438 Builder Work became almost entirely unreachable with 504 errors when the Amazon CloudFront VPC Origins feature failed
    ANDPAD2026
  2. IL-0084 Packet drops on a bare metal provider's vSwitch forced Namespace to rebuild its fra3 region and left global capacity short, and the next day slow NVMe writes on a control plane node drove the Kubernetes API server to very high CPU
    Namespace2024
  3. IL-0145 A power sag and chiller failure in one Australia East availability zone took down the AKS control plane's compute and its non-zone-redundant SQL Database operation queue, blocking AKS create, update and delete operations in Australia East and Australia Southeast
    Microsoft Azure2023
  4. IL-0096 A GKE node pool pinned to the europe-west2-b zone could not scale for 6.5 hours because Google ran out of resources in that zone (ZONE_RESOURCE_POOL_EXHAUSTED)
    2i2c2022
  5. IL-0514 PyPI's backends stopped responding for 7 minutes as rescheduled pods could not start while Consul and Vault were down
    Python Software Foundation2022
  6. IL-0172 An AWS Tokyo data centre cooling failure took down 6 of 60 CockroachDB nodes on Kubernetes within 22 minutes, and ranges lost quorum because the Helm-deployed cluster had no availability zone locality configured
    Devsisters2021
  7. IL-0193 A GCP persistent disk degradation took down a Cassandra StatefulSet whose OrderedReady policy blocked recovery, and Metrictank could not reconnect to the recreated pods, causing a 23-hour Graphite query outage
    Grafana Labs2019
  8. IL-0014 A GKE upgrade to a withdrawn version stalled on slow drains and a stuck node pool, then broke an ingress and confused the HPA
    loveholidays2019
  9. IL-0469 The managed Azure Kubernetes API server for the data100 hub stopped answering, so the stopped hub pod was not recreated
    UC Berkeley2019

Configuration change24 cases

Component
Cloud load balancer6Node pool6Configuration templating5Terraform4Kubernetes API server3Argo CD2CNI plugin2Ingress controller2kubelet2Node launch template2Container memory limit2Persistent volumes and stateful sets2Telemetry agent2AWS VPC CNI2Admission webhook1In-memory cache1CDN and edge network1cert-manager1Configuration management tool1ConfigMaps and Secrets1Container runtime1Coordination and key-value store1Database proxy and connection pooler1Client-side DNS resolver and cache1Authoritative DNS servers and records1etcd1GKE control plane1Jobs and CronJobs1Karpenter1Linux kernel1Node operating system image1Pod priority and preemption1Pod termination1Service discovery1
Contributing factor
Monitoring blind spot7Unverified assumption6Automation amplification6Change with a delayed effect6Unrepresentative staging5Excessive blast radius4Unreviewed defaults4No configuration tests4No runbook4No staged rollout4Missed in review4No safeguard on destructive change3Retry storm3Ignored early signal2Limited debug access2Harmful manual action2No health probes2No headroom2Signal buried in noise1Untested backup1Configuration not in code1Cache-masked failure1Unknown limit1Overlapping failures1Partial rollout1Shared credential1Single point of failure1Unpinned version1
Organisation
DFDS4Grafana Labs3OpenAI2AdGuard VPN1CoW Protocol1DigitalOcean1Discord1InfluxData1Jenkins project1Jetstack1MIT Open Learning1Pusher1Python Software Foundation1Skyscanner1Stytch1Target1UK Ministry of Justice1Wikimedia Foundation1
  1. IL-0512 xpro.mit.edu returned 503 for an hour when its Kubernetes/APISIX origin certificate expired after renewal failed
    MIT Open Learning2026
  2. IL-0499 Discord voice and video calls hung at Awaiting Endpoint for three hours after 17% of sessions stopped at once
    Discord2026
  3. IL-0063 VPN outage after recreating a Redis StatefulSet restarted the whole Redis cluster without readiness checks
    AdGuard VPN2025
  4. IL-0192 A Crossplane composition merge-policy change for a TLS policy update switched Kubernetes Services from LoadBalancer to ClusterIP and deleted the load balancers behind about 25% of Grafana Cloud services
    Grafana Labs2025
  5. IL-0005 A new telemetry service overloaded the Kubernetes API servers of large clusters and broke DNS-based service discovery for all OpenAI services
    OpenAI2024
  6. IL-0114 An unpinned AWS VPC CNI plugin was updated automatically during a capacity expansion, leaving new pods Pending without IP addresses and stopping CoW Protocol settlements for about 3.5 hours
    CoW Protocol2024
  7. IL-0153 DOKS worker nodes went NotReady at 06:00 UTC on three days because a systemd timer in the Debian image ran apt upgrade and killed it mid kernel upgrade, breaking node networking
    DigitalOcean2023
  8. IL-0473 Jenkins infrastructure went fully down after a pod IP range fix deleted every AKS node pool and could not recreate them
    Jenkins project2023
  9. IL-0091 Changing the worker node instance type through Terraform recycled all nodes at once and deleted users' production pods together
    UK Ministry of Justice2023
  10. IL-0231 Deleting EKS managed node groups also deleted Karpenter's instance profile, and every Stytch worker node shut down
    Stytch2023
  11. IL-0109 A retried node rollover on a DFDS EKS cluster set the Auto Scaling group back to 15 instances, and the default termination policy removed every healthy worker node, leaving the cluster with no nodes
    DFDS2023
  12. IL-0116 A CNI configuration change incompatible with a hardware variant that exists only in production cut network connectivity in an OpenAI GPU cluster and took text-davinci-003 fully down
    OpenAI2023
  13. IL-0513 PyPI backends on Kubernetes were unreachable over HTTPS for 20 minutes when the Ingress ELB lost its HTTPS listener
    Python Software Foundation2022
  14. IL-0191 A label selector accidentally added to the Loki distributor Service matched only the new pods of a rollout, which took all write traffic, were OOM-killed and stalled the rollout
    Grafana Labs2022
  15. IL-0009 A duplicated generated name made Argo CD replace the InfluxDB Cloud production Application and delete the whole production environment
    InfluxData2022
  16. IL-0111 Switching the main Kubernetes ingress controller from Traefik v1 to v2 stopped X-Forwarded headers being passed, which broke the Okta login flow for every service using Okta and stopped DFDS online bookings for over an hour
    DFDS2021
  17. IL-0010 A templating change that dropped its {{ }} delimiters made Argo CD delete all 478 Skyscanner services in every region
    Skyscanner2021
  18. IL-0105 Replacing the AWS VPC CNI during a Terraform refactoring detached the ALB target groups, and every service in the Kubernetes cluster returned 503
    DFDS2021
  19. IL-0365 A Terraform change to auto-scaling groups detached all worker nodes from both ALBs and cut off every cluster service
    DFDS2020
  20. IL-0458 Toolforge Kubernetes API server failed to start after a restart, logging "etcd cluster is unavailable or misconfigured"
    Wikimedia Foundation2019
  21. IL-0008 An unresponsive OPA validating webhook crash-looped the API server during a GKE master upgrade, and node auto-repair then recreated every node
    Jetstack2019
  22. IL-0004 A new default pod priority let a new Cortex cluster preempt every production ingester in a cascade, taking Grafana Cloud Hosted Prometheus down
    Grafana Labs2019
  23. IL-0066 Broken ConfigMap change lay dormant for weeks, then took down Pusher's edge proxy when a cluster upgrade replaced its pods
    Pusher2019
  24. IL-0055 A failing CronJob with the default concurrency policy piled up thousands of restarting pods and took down a cluster
    Target2018

Faulty deployment6 cases

Component
Node pool3Ingress controller2Calico1CI system1Cloud load balancer1Cluster autoscaler1Descheduler1Model inference server1Pod termination1Service discovery1Telemetry agent1
Contributing factor
Unverified assumption3Excessive blast radius2Unreviewed defaults2Monitoring blind spot2No alert on limit2Change started before off-hours1Change with a delayed effect1No runbook1No headroom1No staged rollout1Overlapping failures1Single point of failure1
Organisation
Airbnb1CircleCI1Kubernetes project1Lucidworks1UK Ministry of Justice1Yonghyun Kim1
  1. IL-0237 A Kubernetes patch upgrade evicted every Ray Serve head and worker pod at once, and Lucidworks hosted LLMs went down
    Lucidworks2026
  2. IL-0188 Rolling updates on EKS returned 502 errors for about 30 seconds because Kubernetes deleted old pods as soon as new pods passed readiness, before the ALB had marked the new pod targets healthy
    Yonghyun Kim2025
  3. IL-0020 During an EKS 1.30 upgrade the descheduler tried to reschedule every pod on the cordoned nodes at once and production workloads went down
    UK Ministry of Justice2025
  4. IL-0086 A deployment meant for a CircleCI test Kubernetes cluster also removed Traefik and Calico resources from the production cluster, taking down the CircleCI UI and runner.circleci.com
    CircleCI2021
  5. IL-0034 A crashing stats agent sidecar marked every pod unready and the service mesh pulled all services out of traffic
    Airbnb2019
  6. IL-0467 Boskos crash-looped after a node drain started it from a buggy StatefulSet image, failing Kubernetes PR jobs for 9 hours
    Kubernetes project2019

Breaking change in an upgrade12 cases

Component
Admission webhook4Ingress controller4Node operating system image4Cloud load balancer3ConfigMaps and Secrets2Client-side DNS resolver and cache2GKE control plane2Helm2kube-proxy2Calico1CDN and edge network1cert-manager1Cilium1CoreDNS1etcd1Istio1Karpenter1Kubernetes API server1kubelet1Linux kernel1Node pool1Persistent volumes and stateful sets1systemd-networkd1Web application firewall rules1
Contributing factor
Monitoring blind spot5Change with a delayed effect3Unrepresentative staging3Harmful manual action2No staged rollout2Ineffective rollback2Single point of failure2Deferred upgrades2Unverified assumption1Automation amplification1Untested backup1Configuration not in code1Unreviewed defaults1Cache-masked failure1Unpatched known bug1Limited debug access1No headroom1Partial rollout1Unpinned version1
Organisation
PortSwigger22i2c1Altinity1CircleCI1Datadog1Honeycomb1Jenkins project1Reddit1Sunwoo Park1UK Ministry of Justice1Wealthy1
  1. IL-0092 Upgrading ingress-nginx from 1.12.0 to 1.14.3 brought a newer ModSecurity and OWASP Core Rule Set whose false positives blocked HMPPS Auth, so users could not sign in to their applications
    UK Ministry of Justice2026
  2. IL-0405 An expired AWS Load Balancer Controller webhook certificate left no healthy targets and took BSEE SaaS down
    PortSwigger2026
  3. IL-0101 After Google forced a GKE control plane upgrade to 1.33.2 at a deprecation deadline, no new node could initialise, so autoscaling and spot node replacement stopped for 15 hours
    Wealthy2025
  4. IL-0079 A GKE node auto-upgrade to an ubuntu_containerd image with a kernel regression left kube-proxy unable to apply its ip6tables rules, so load balancer health checks failed and several clusters lost all ingress traffic
    Altinity2025
  5. IL-0404 All BSEE SaaS customer pods on EKS crash-looped after Karpenter moved nodes to Bottlerocket v1.26.0, which broke Java
    PortSwigger2024
  6. IL-0203 A routine Kubernetes cluster system update made the AWS Load Balancer Controller fail, so new pods were never registered with the ALB and Honeycomb's EU1 ingest and UI went down once the last old pods were replaced
    Honeycomb2024
  7. IL-0180 A GKE control plane auto-upgrade from 1.22 to 1.23 removed the v1beta1 Ingress and ValidatingWebhookConfiguration APIs that Istio 1.7.4 still used, and Istio stopped working in production
    Sunwoo Park2023
  8. IL-0060 Kubernetes 1.24 upgrade removed the master node label that Calico route reflectors selected on, dropping all pod routes and taking Reddit down for 314 minutes
    Reddit2023
  9. IL-0230 kube-proxy sync failures during a Kubernetes upgrade corrupted node iptables, and CircleCI jobs stopped starting
    CircleCI2023
  10. IL-0198 An automatic Ubuntu security update restarted systemd-networkd, which flushed the IP rules Cilium relied on and cut more than 60 percent of Datadog's Kubernetes nodes off the network across regions and cloud providers
    Datadog2023
  11. IL-0501 Every node of the UToronto Kubernetes cluster on Azure went NotReady and the JupyterHub became unreachable
    2i2c2022
  12. IL-0522 An ingress-nginx chart upgrade deleted the jenkins.io Ingress and Fastly served HTTP/503 errors for hours
    Jenkins project2021

Operator error7 cases

Component
Ingress controller4Database2Persistent volumes and stateful sets2Terraform2CI system1Cloud load balancer1Credential broker for pods1external-dns1Jobs and CronJobs1JupyterHub1kube-proxy1Model hub1Node pool1Prometheus1VPC route tables and firewall rules1
Contributing factor
No safeguard on destructive change6Unverified assumption1Unreviewed defaults1Harmful manual action1No runbook1No headroom1Overlapping failures1Deferred upgrades1Unsafe failure behaviour1
Organisation
UK Ministry of Justice22i2c1DevOps Hof1GitLab1OpenAI1UC Berkeley1
  1. IL-0095 An engineer deleted the prod namespace of the LEAP JupyterHub while believing the kubeconfig pointed at a new cluster, and the hub was down for about an hour
    2i2c2025
  2. IL-0244 A queued pull request destroyed the EKS subnet route table associations and Cloud Platform services went offline
    UK Ministry of Justice2024
  3. IL-0207 A routine task run from an engineer's local environment switched to production and deleted GitLab.com's Consul release, so Patroni lost Consul and GitLab.com returned 5xx errors to all traffic
    GitLab2022
  4. IL-0115 One of OpenAI's three production AKS clusters went offline through a human error, and rebuilding it took 10½ hours because an AKS Kubernetes upgrade broke the ingress and AKS cannot be downgraded
    OpenAI2022
  5. IL-0089 A test cluster deletion script run with the kube context pointing at the live-1 production cluster deleted its ingress controller, monitoring, logging, cert-manager, kiam and external-dns
    UK Ministry of Justice2020
  6. IL-0054 All ingress traffic lost while draining a GKE node pool because the load balancer only targeted new nodes without ingress-nginx pods
    DevOps Hof2019
  7. IL-0506 UC Berkeley DataHub logins failed and students got empty home directories after the JupyterHub database was overwritten
    UC Berkeley2017

Software defect11 cases

Component
Kubernetes API server3etcd2Ingress controller2kubelet2Persistent volumes and stateful sets2Cloud load balancer1Cluster autoscaler1Database1GKE control plane1GPU driver1JupyterHub1Linkerd1Linux kernel1Container memory limit1Node pool1Telemetry agent1Terraform1
Contributing factor
Monitoring blind spot3Single point of failure3Unverified assumption2Change with a delayed effect2Excessive blast radius1Unreviewed defaults1Ignored early signal1Unpatched known bug1Harmful manual action1No alert on limit1No configuration tests1No staged rollout1Overlapping failures1Ineffective rollback1Deferred upgrades1Unrepresentative staging1
Organisation
Binder Project2AirMap1Barush Mendez1DFDS1Linode1Monzo1Nordstrom1Palark1thredUP1ZenML1
  1. IL-0078 Enabling EKS secrets encryption with KMS through Terraform removed worker node IAM permissions and took down all ZenML Pro workspaces
    ZenML2025
  2. IL-0157 After a release, new nodes could not join Linode Kubernetes Engine clusters with IP ACL enabled, so autoscaling failed on about 20 to 25 clusters
    Linode2025
  3. IL-0138 A failed Nvidia driver build left broken kernel packages on the only Kubernetes master node, which hit a kernel panic on reboot after a power outage and took the homelab cluster down for six days
    Barush Mendez2025
  4. IL-0403 mybinder.org could not create pods for hours as the GKE gke-resource-quotas quota wrongly reported 15k pods in use
    Binder Project2022
  5. IL-0106 A malformed Traefik IngressRoute triggered a Traefik v2 bug once a node rollover restarted every Traefik pod, and all Kubernetes services returned 404
    DFDS2021
  6. IL-0196 Adding a second Redis replica and lowering memory limits in one RedisFailover change restarted the only Redis pod that held data, and Sentinel promoted the empty new replica, erasing the data
    Palark2020
  7. IL-0051 A third-party log shipping client leaked file descriptors with every new container until nodes failed with too many open files
    thredUP2018
  8. IL-0074 All AirMap platform services down as worker nodes went NotReady and pods crash-looped on network and DNS failures
    AirMap2018
  9. IL-0504 mybinder.org launches failed for about five hours as undeleted user pods piled up and filled the cluster
    Binder Project2018
  10. IL-0072 High etcd write latency made the API servers return stale data, and the production cluster flipped between two views of its state
    Nordstrom2017
  11. IL-0003 linkerd lost Kubernetes service discovery after an etcd reconfiguration, and restarting every linkerd instance turned failed payments into a full platform outage
    Monzo2017

unknown3 cases

Component
Calico1Container runtime1etcd1Kubernetes API server1kubelet1Multus meta-CNI1Node pool1Persistent volumes and stateful sets1
Contributing factor
Harmful manual action2Unverified assumption1Unreviewed defaults1Ignored early signal1Monitoring blind spot1Pages not delivered1Single point of failure1
Organisation
Binder Project1Fyra Labs1NRE Labs1
  1. IL-0087 After a cold reboot of RKE2 nodes, pods could not get IP addresses because Multus could not reach the Kubernetes API, and further reboots spread the failure to every node
    Fyra Labs2023
  2. IL-0059 Single-master Kubernetes control plane kept dying because the kubelet restarted a healthy etcd on a failing kubeadm liveness probe
    NRE Labs2018
  3. IL-0402 mybinder.org launches dropped to zero for hours because a JupyterHub pod on a lost GKE node kept its hub database PVC
    Binder Project2018

Certificate or credential expiry2 cases

Component
cert-manager1GKE control plane1Kubernetes API server1Node pool1RBAC and service accounts1Secrets store1
Contributing factor
Single point of failure2Unverified assumption1Limited debug access1Harmful manual action1Monitoring blind spot1
Organisation
GoCardless1Webhook Relay1
  1. IL-0113 The managed GKE control plane of Webhook Relay's main cluster failed because its API server certificates had never been rotated, and a manual credential rotation reported success without taking effect
    Webhook Relay2022
  2. IL-0460 Vault served an expired TLS certificate and almost every GoCardless service pod crash-looped, a 22-minute full outage
    GoCardless2020

DNS or network fault2 cases

Component
Container registry and its credentials2CoreDNS1Container memory limit1Object storage1
Contributing factor
Single point of failure2Harmful manual action1Overlapping failures1
Organisation
GitHub1Gitpod1
  1. IL-0517 Gitpod workspaces failed to start worldwide and lost data when CoreDNS lookups to 8.8.8.8 timed out in its clusters
    Gitpod2022
  2. IL-0199 A non-essential container exceeding its memory limit made GitHub's production pods unavailable, and their restarts failed because a DNS maintenance change had left the clusters unable to reach the image registry
    GitHub2020