Incident Library
Applied filters SymptomService outageDomainKubernetesClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases96 cases

24 to 28 of 96

  1. IL-0157 After a release, new nodes could not join Linode Kubernetes Engine clusters with IP ACL enabled, so autoscaling failed on about 20 to 25 clusters
    Linode2025
    Symptoms

    From about 01:00 UTC on 2025-03-05, Linode Kubernetes Engine (LKE) clusters with IP ACL enabled failed to autoscale: new nodes could not join the cluster, and some customers had service disruptions. About 20 to 25 clusters in several regions around the world were affected; the status page lists the US-IAD (Washington), US-ORD (Chicago), US-SEA (Seattle) and IN-MAA (Chennai) LKE components.

    Scale-up failureService outage
    Trigger

    Akamai deployed the latest LKE software release.

    Root cause
    Software defect
    Component
    Cluster autoscaler
    Contributing factor
    –
    Detection

    Customers began having problems at about 01:00 UTC, and the first status post, at 06:49 UTC, said the team was investigating an LKE autoscaling problem when IP ACL is turned on.

    Remediation
    • restarted the gateway pods, which gave only short-term relief
    • rolled back the release on all affected clusters
    Action items
    • Continue investigating the root cause and take preventive actions
    Lessons

    The defect hit only clusters with IP ACL enabled, and it showed when the autoscaler tried to add nodes. Restarting the gateway pods did not hold; rolling back the release ended the incident.

    View case
  2. IL-0138 A failed Nvidia driver build left broken kernel packages on the only Kubernetes master node, which hit a kernel panic on reboot after a power outage and took the homelab cluster down for six days
    Barush Mendez2025
    Symptoms

    Every service in the homelab Kubernetes cluster stopped answering: the V-Rising server, Grafana, Jellyfin and Linkding were all unreachable, and the Kubernetes API was down, so the cluster could not be managed at all. The outage was found by hand, about 3 hours after it began, when someone tried to use the services.

    Control plane unresponsiveService outageMonitoring outage
    Trigger

    A power outage forced the Kubernetes master node to reboot.

    Root cause
    Software defect
    Component
    GPU driverLinux kernelKubernetes API server
    Contributing factor
    Ignored early signalChange with a delayed effectSingle point of failureMonitoring blind spot
    Detection

    Manually, when someone tried to access the services; there were no automated alerts.

    Remediation
    • Connected a monitor to the master node and read the boot logs, which showed the kernel panic
    • Booted the previous kernel version, which brought the system back
    • Removed the broken Nvidia packages with sudo apt-get remove --purge '^nvidia-.*' and ran sudo dpkg --configure -a
    • Found remaining broken packages with dpkg -l | grep -v ^ii and removed them by hand
    • Ran sudo apt-get install -f and reinstalled linux-headers-$(uname -r)
    • Rebooted and confirmed the kernel panic was gone
    Action items
    • Make sure GPU driver installations fully succeed, or revert immediately when they fail
    • Move the cluster to a high-availability control plane
    • Expose the Kubernetes API through a load balancer in the HA setup
    • Set up a ticketing system to record incidents and their resolution
    • Write predefined troubleshooting workflows to shorten response time
    • Add infrastructure monitoring such as Prometheus and Grafana
    • Add blackbox HTTP and TCP checks from outside the cluster for critical services
    Lessons

    A driver installation that fails half-way can leave kernel packages broken without any visible effect until the next reboot, which may come weeks later and at the worst time. A failed DKMS build should be fixed or reverted at once. With a single master node and no monitoring from outside the cluster, one unbootable machine meant a total outage that nobody was told about; the fix itself was simple once the boot logs were seen.

    View case
  3. IL-0192 A Crossplane composition merge-policy change for a TLS policy update switched Kubernetes Services from LoadBalancer to ClusterIP and deleted the load balancers behind about 25% of Grafana Cloud services
    Grafana Labs2025
    Symptoms

    At 08:41 UTC on 18 February 2025, several Grafana Labs application teams noticed that some of their endpoints were no longer available. The investigation quickly found that the load balancers for those endpoints were gone.

    Service outageResources deleted
    Trigger

    A change to the merge policy of a Crossplane composition, made so that the default TLS policy of the load balancers could be overridden, was applied to development, staging and production at the same time.

    Root cause
    Configuration change
    Component
    Configuration templatingCloud load balancer
    Contributing factor
    Unverified assumptionNo staged rolloutExcessive blast radiusNo safeguard on destructive change
    Detection

    Application teams noticed at 08:41 UTC that some of their endpoints were no longer available.

    Remediation
    • rolled back the Crossplane composition change
    • recreated the affected Kubernetes Services so that Kubernetes recreated their load balancers
    Action items
    • Add a CI check that blocks a deployment to more than one wave, so any exception to the dev, staging, canary, production order is caught
    • Give higher priority to a more sophisticated wave design that was already under discussion
    • Enable deletion protection on the load balancers, one layer below Crossplane
    • Enable deletion protection on other critical cloud provider resources
    • Develop the Crossplane Managed Resources exporter for change validation and visibility, now in beta testing within Platform engineering; simulated against the change that caused this outage, it shows a clear diff
    • Complete and test support for reconfiguring the TLS policy so application teams can deploy it
    Lessons

    A change that looked small and low-risk had a side effect that only a full end-to-end test through Crossplane and the cloud provider's load balancer controller would have shown; seeing the intended override work was not that test. Staged rollout and blast-radius limits that exist only as process can be skipped, so Grafana Labs moved to enforce waves in CI and to add deletion protection on the load balancers themselves, beneath the Crossplane layer.

    View case
  4. IL-0489 Rejoining a failed node of Magic Pages' three-node MySQL cluster broke its metadata and took customer websites down
    Magic Pages2025
    Symptoms

    On 15 January 2025, from about 09:55 to 11:36 CET, a database outage at Magic Pages took down customer websites. It began when the founder tried to reconnect a failed third node to the high-availability MySQL cluster and the database server reported ambiguous errors about missing metadata information.

    Service outage
    Trigger

    One of the three nodes of the MySQL cluster went down, and reconnecting it failed with errors about missing metadata information.

    Root cause
    Dependency failure
    Component
    DatabasePersistent volumes and stateful sets
    Contributing factor
    Harmful manual actionSingle point of failureMonitoring blind spot
    Remediation
    • Restarted the storage server to fix PVC provisioning
    • Let the MySQL nodes restore and sync with each other for about 45 minutes to rebuild the cluster metadata
    • Cleaned up the corrupted cluster metadata
    • Redeployed MySQL Router
    Action items
    • Monitor the health of the storage system more closely
    • Improve the failover procedures for MySQL Router
    • Investigate and fix the status page so it shows reliable information
    Lessons

    The cluster survived the first node failure as designed and lost no data, but its metadata turned out to be a single point of failure. Better monitoring of PVC health could help prevent similar issues, the 45-minute recovery leaves room for improvement, and the status page has to detect failing customer sites reliably.

    View case
  5. IL-0005 A new telemetry service overloaded the Kubernetes API servers of large clusters and broke DNS-based service discovery for all OpenAI services
    OpenAI2024
    Symptoms

    Alerts fired at 3:13 PM PST on 11 December 2024, and from 3:16 PM every OpenAI service was significantly degraded or completely unavailable, including ChatGPT, the API and Sora. The status page reported API calls returning errors and difficulties logging in to platform.openai.com and ChatGPT. Customer impact peaked at 3:40 PM.

    Service outageHigh error rateControl plane unresponsiveDNS resolution failures
    Trigger

    At 2:23 PM PST a change introducing a new telemetry service, which collects detailed Kubernetes control plane metrics, was merged and the deployment pipeline started. Between 2:51 PM and 3:20 PM the change was applied to all clusters.

    Root cause
    Configuration change
    Component
    Kubernetes API serverClient-side DNS resolver and cacheTelemetry agent
    Contributing factor
    Unrepresentative stagingNo staged rolloutCache-masked failureMonitoring blind spotLimited debug access
    Detection

    OpenAI's detection tools worked: alerts fired at 3:13 PM PST, a few minutes before customer impact began at 3:16 PM, and the root cause was identified at 3:16 PM.

    Remediation
    • Scaled clusters down to reduce the total Kubernetes API load
    • Blocked network access to the Kubernetes admin APIs so no new expensive requests arrived
    • Scaled up the Kubernetes API servers
    • Removed the telemetry service once enough control was back
    • Moved traffic from impacted clusters to healthy ones
    • Intervened manually in clusters where simultaneous resource downloads saturated resource limits
    Action items
    • Put every infrastructure configuration change through a phased rollout with continuous monitoring of both the service workloads and the clusters, including the Kubernetes control plane
    • Run fault injection tests in which the data plane runs without the control plane for longer, and tests that roll out bad changes on purpose to check detection and rollback
    • Build break-glass mechanisms so engineers can reach the Kubernetes API servers in any circumstances
    • Decouple the Kubernetes data plane from the control plane so that service discovery for critical workloads does not depend on the control plane
    • Speed up recovery with better caching and dynamic rate limiters for resources needed at cluster startup, and regular drills that rapidly replace an entire cluster
    Lessons

    Several systems and processes failed at once and interacted in unexpected ways: tests did not catch the change's effect on the Kubernetes control plane, DNS caching put a delay between the change and the failures, and being locked out of the control plane made remediation very slow. Because service discovery used Kubernetes DNS, the control plane was carrying load for product traffic even though the data plane was meant to be independent of it.

    View case
Check your cluster for this

These 96 cases involve Node pool and Ingress controller. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause11

Capacity exhaustion17 cases

Component
Database4Container memory limit4Node pool4CoreDNS3Persistent volumes and stateful sets3In-memory cache2Helm2Ingress controller2kubelet2Linux kernel2Pod scheduling and placement2Admission webhook1BGP routing1Cloud block storage1Cloud controller manager1CNI plugin1conntrack table1Container runtime1Client-side DNS resolver and cache1Horizontal pod autoscaler1InfiniBand fabric1Istio1Jobs and CronJobs1JupyterHub1kube-dns1Model hub1NAT gateway1Network control plane (SDN)1NIC, cable and switch port link1Node root disk1Container registry and its credentials1
Contributing factor
Single point of failure7Automation amplification6Monitoring blind spot5Unverified assumption4Excessive blast radius4Unreviewed defaults4Retry storm3Change during peak hours2Unknown limit2Harmful manual action2No caching2No resource requests or limits2Ignored early signal1Limited debug access1No alert on limit1No capacity planning1No runbook1No headroom1Unrepresentative staging1
Organisation
Backpackr22i2c1BC Government1Binder Project1Buildkite1Datadog1freedesktop.org1Honeycomb1IONOS Cloud1Magic Pages1Mercari1Moonlight1RebelMouse1Saltside1Tinder1Universe1
  1. IL-0102 CoreDNS pods on a production EKS cluster ran out of memory under a burst of new pods, endpoints and nodes, taking down the Buildkite web UI, APIs and job dispatch
    Buildkite2026
  2. IL-0350 Hosts in an IONOS TXL cluster lost connectivity when an InfiniBand fabric resweep failed on a full multicast table
    IONOS Cloud2026
  3. IL-0067 Redis filled its persistent volume, corrupted its append-only file and crash-looped on every restart
    BC Government2025
  4. IL-0080 kube-dns on Mercari's GKE clusters could not keep up with peak-hour DNS query rates, and a Sev1 production outage followed when several services failed DNS resolution
    Mercari2025
  5. IL-0082 Losing one CoreDNS pod during a planned EKS node reroll overloaded the remaining CoreDNS pods, causing DNS timeouts and errors across multiple services at RebelMouse
    RebelMouse2025
  6. IL-0170 During a Lunar New Year event, traffic more than three times normal filled the nf_conntrack table on the EKS node running the Istio ingress gateway, dropping packets and taking down Backpackr's web and app services
    Backpackr2025
  7. IL-0027 All CoreDNS pods landed on one node, the node ran out of memory, and DNS for the whole service went down
    Backpackr2025
  8. IL-0085 A control plane node of Magic Pages' Kubernetes cluster on Hetzner Cloud ran out of memory, the cluster's own calls then used up the Hetzner API rate limit, and all customer websites were down for about 7.5 hours
    Magic Pages2025
  9. IL-0202 A burst of queries and database writes exhausted connection pools and memory on Honeycomb's ingest hosts, and Kubernetes crash-loop back-off kept failed hosts offline so ingest capacity stayed too low to recover
    Honeycomb2023
  10. IL-0094 Student pods filled the shared nodes of a JupyterHub cluster and starved the hub's core pods, freezing the hub
    2i2c2020
  11. IL-0037 Image registry rate-limited a whole region after a crash-looping DaemonSet with imagePullPolicy Always pulled through three NAT instances
    Datadog2019
  12. IL-0064 CPU-heavy pods packed onto one Google Cloud node drove it to kernel panics, and the crash moved from node to node until the site was down
    Moonlight2019
  13. IL-0472 Node-to-pod traffic and DNS failed on Tinder's 605-node Flannel cluster when every node's ARP cache passed gc_thresh3
    Tinder2019
  14. IL-0518 gitlab.freedesktop.org on GKE ran short of volume space and a planned 30-minute storage upgrade became a 4-hour outage
    freedesktop.org2018
  15. IL-0503 mybinder.org launches failed after a GKE node ran out of memory and its nginx-ingress proxy pod crash-looped
    Binder Project2017
  16. IL-0036 A reindex Job that kept failing exhausted one node after another until the whole cluster was down
    Universe2017
  17. IL-0040 Parallel Helm installs saturated small gp2 root volumes on a kops cluster and left all three workers NotReady with the kubelet or Docker down
    Saltside2017

Dependency failure3 cases

Component
Cloud block storage1Database1Data center power and cooling1etcd1Liveness and readiness probes1InfiniBand fabric1Persistent volumes and stateful sets1
Contributing factor
Monitoring blind spot2Single point of failure2Automation amplification1Excessive blast radius1Unreviewed defaults1Harmful manual action1No alert on limit1No runbook1Overlapping failures1
Organisation
Magic Pages1Nebius1Yonghyun Kim1
  1. IL-0133 A storm shut down cooling at Nebius's us-central1 data center, and a disk hot-plug bug kept Managed Kubernetes control planes down for hours after power returned
    Nebius2026
  2. IL-0187 A brief Spring Cloud Config Server downtime failed every client application's liveness probe, because the actuator health check included the config server, and Kubernetes restarted the pods in a kill loop
    Yonghyun Kim2025
  3. IL-0489 Rejoining a failed node of Magic Pages' three-node MySQL cluster broke its metadata and took customer websites down
    Magic Pages2025

Cloud provider outage9 cases

Component
Database3AKS control plane2Cluster autoscaler2Ingress controller2Kubernetes API server2Node pool2CDN and edge network1Cloud load balancer1Coordination and key-value store1GKE control plane1Helm1Horizontal pod autoscaler1kubelet1Node root disk1Persistent volumes and stateful sets1Secrets store1Web application firewall rules1
Contributing factor
Overlapping failures3Single point of failure3Unverified assumption2Excessive blast radius2Automation amplification1Ignored early signal1Unpatched known bug1Limited debug access1No capacity planning1No headroom1Pages not delivered1Unrepresentative staging1
Organisation
2i2c1ANDPAD1Devsisters1Grafana Labs1loveholidays1Microsoft Azure1Namespace1Python Software Foundation1UC Berkeley1
  1. IL-0438 Builder Work became almost entirely unreachable with 504 errors when the Amazon CloudFront VPC Origins feature failed
    ANDPAD2026
  2. IL-0084 Packet drops on a bare metal provider's vSwitch forced Namespace to rebuild its fra3 region and left global capacity short, and the next day slow NVMe writes on a control plane node drove the Kubernetes API server to very high CPU
    Namespace2024
  3. IL-0145 A power sag and chiller failure in one Australia East availability zone took down the AKS control plane's compute and its non-zone-redundant SQL Database operation queue, blocking AKS create, update and delete operations in Australia East and Australia Southeast
    Microsoft Azure2023
  4. IL-0096 A GKE node pool pinned to the europe-west2-b zone could not scale for 6.5 hours because Google ran out of resources in that zone (ZONE_RESOURCE_POOL_EXHAUSTED)
    2i2c2022
  5. IL-0514 PyPI's backends stopped responding for 7 minutes as rescheduled pods could not start while Consul and Vault were down
    Python Software Foundation2022
  6. IL-0172 An AWS Tokyo data centre cooling failure took down 6 of 60 CockroachDB nodes on Kubernetes within 22 minutes, and ranges lost quorum because the Helm-deployed cluster had no availability zone locality configured
    Devsisters2021
  7. IL-0193 A GCP persistent disk degradation took down a Cassandra StatefulSet whose OrderedReady policy blocked recovery, and Metrictank could not reconnect to the recreated pods, causing a 23-hour Graphite query outage
    Grafana Labs2019
  8. IL-0014 A GKE upgrade to a withdrawn version stalled on slow drains and a stuck node pool, then broke an ingress and confused the HPA
    loveholidays2019
  9. IL-0469 The managed Azure Kubernetes API server for the data100 hub stopped answering, so the stopped hub pod was not recreated
    UC Berkeley2019

Configuration change24 cases

Component
Cloud load balancer6Node pool6Configuration templating5Terraform4Kubernetes API server3Argo CD2CNI plugin2Ingress controller2kubelet2Node launch template2Container memory limit2Persistent volumes and stateful sets2Telemetry agent2AWS VPC CNI2Admission webhook1In-memory cache1CDN and edge network1cert-manager1Configuration management tool1ConfigMaps and Secrets1Container runtime1Coordination and key-value store1Database proxy and connection pooler1Client-side DNS resolver and cache1Authoritative DNS servers and records1etcd1GKE control plane1Jobs and CronJobs1Karpenter1Linux kernel1Node operating system image1Pod priority and preemption1Pod termination1Service discovery1
Contributing factor
Monitoring blind spot7Unverified assumption6Automation amplification6Change with a delayed effect6Unrepresentative staging5Excessive blast radius4Unreviewed defaults4No configuration tests4No runbook4No staged rollout4Missed in review4No safeguard on destructive change3Retry storm3Ignored early signal2Limited debug access2Harmful manual action2No health probes2No headroom2Signal buried in noise1Untested backup1Configuration not in code1Cache-masked failure1Unknown limit1Overlapping failures1Partial rollout1Shared credential1Single point of failure1Unpinned version1
Organisation
DFDS4Grafana Labs3OpenAI2AdGuard VPN1CoW Protocol1DigitalOcean1Discord1InfluxData1Jenkins project1Jetstack1MIT Open Learning1Pusher1Python Software Foundation1Skyscanner1Stytch1Target1UK Ministry of Justice1Wikimedia Foundation1
  1. IL-0512 xpro.mit.edu returned 503 for an hour when its Kubernetes/APISIX origin certificate expired after renewal failed
    MIT Open Learning2026
  2. IL-0499 Discord voice and video calls hung at Awaiting Endpoint for three hours after 17% of sessions stopped at once
    Discord2026
  3. IL-0063 VPN outage after recreating a Redis StatefulSet restarted the whole Redis cluster without readiness checks
    AdGuard VPN2025
  4. IL-0192 A Crossplane composition merge-policy change for a TLS policy update switched Kubernetes Services from LoadBalancer to ClusterIP and deleted the load balancers behind about 25% of Grafana Cloud services
    Grafana Labs2025
  5. IL-0005 A new telemetry service overloaded the Kubernetes API servers of large clusters and broke DNS-based service discovery for all OpenAI services
    OpenAI2024
  6. IL-0114 An unpinned AWS VPC CNI plugin was updated automatically during a capacity expansion, leaving new pods Pending without IP addresses and stopping CoW Protocol settlements for about 3.5 hours
    CoW Protocol2024
  7. IL-0153 DOKS worker nodes went NotReady at 06:00 UTC on three days because a systemd timer in the Debian image ran apt upgrade and killed it mid kernel upgrade, breaking node networking
    DigitalOcean2023
  8. IL-0473 Jenkins infrastructure went fully down after a pod IP range fix deleted every AKS node pool and could not recreate them
    Jenkins project2023
  9. IL-0091 Changing the worker node instance type through Terraform recycled all nodes at once and deleted users' production pods together
    UK Ministry of Justice2023
  10. IL-0231 Deleting EKS managed node groups also deleted Karpenter's instance profile, and every Stytch worker node shut down
    Stytch2023
  11. IL-0109 A retried node rollover on a DFDS EKS cluster set the Auto Scaling group back to 15 instances, and the default termination policy removed every healthy worker node, leaving the cluster with no nodes
    DFDS2023
  12. IL-0116 A CNI configuration change incompatible with a hardware variant that exists only in production cut network connectivity in an OpenAI GPU cluster and took text-davinci-003 fully down
    OpenAI2023
  13. IL-0513 PyPI backends on Kubernetes were unreachable over HTTPS for 20 minutes when the Ingress ELB lost its HTTPS listener
    Python Software Foundation2022
  14. IL-0191 A label selector accidentally added to the Loki distributor Service matched only the new pods of a rollout, which took all write traffic, were OOM-killed and stalled the rollout
    Grafana Labs2022
  15. IL-0009 A duplicated generated name made Argo CD replace the InfluxDB Cloud production Application and delete the whole production environment
    InfluxData2022
  16. IL-0111 Switching the main Kubernetes ingress controller from Traefik v1 to v2 stopped X-Forwarded headers being passed, which broke the Okta login flow for every service using Okta and stopped DFDS online bookings for over an hour
    DFDS2021
  17. IL-0010 A templating change that dropped its {{ }} delimiters made Argo CD delete all 478 Skyscanner services in every region
    Skyscanner2021
  18. IL-0105 Replacing the AWS VPC CNI during a Terraform refactoring detached the ALB target groups, and every service in the Kubernetes cluster returned 503
    DFDS2021
  19. IL-0365 A Terraform change to auto-scaling groups detached all worker nodes from both ALBs and cut off every cluster service
    DFDS2020
  20. IL-0458 Toolforge Kubernetes API server failed to start after a restart, logging "etcd cluster is unavailable or misconfigured"
    Wikimedia Foundation2019
  21. IL-0008 An unresponsive OPA validating webhook crash-looped the API server during a GKE master upgrade, and node auto-repair then recreated every node
    Jetstack2019
  22. IL-0004 A new default pod priority let a new Cortex cluster preempt every production ingester in a cascade, taking Grafana Cloud Hosted Prometheus down
    Grafana Labs2019
  23. IL-0066 Broken ConfigMap change lay dormant for weeks, then took down Pusher's edge proxy when a cluster upgrade replaced its pods
    Pusher2019
  24. IL-0055 A failing CronJob with the default concurrency policy piled up thousands of restarting pods and took down a cluster
    Target2018

Faulty deployment6 cases

Component
Node pool3Ingress controller2Calico1CI system1Cloud load balancer1Cluster autoscaler1Descheduler1Model inference server1Pod termination1Service discovery1Telemetry agent1
Contributing factor
Unverified assumption3Excessive blast radius2Unreviewed defaults2Monitoring blind spot2No alert on limit2Change started before off-hours1Change with a delayed effect1No runbook1No headroom1No staged rollout1Overlapping failures1Single point of failure1
Organisation
Airbnb1CircleCI1Kubernetes project1Lucidworks1UK Ministry of Justice1Yonghyun Kim1
  1. IL-0237 A Kubernetes patch upgrade evicted every Ray Serve head and worker pod at once, and Lucidworks hosted LLMs went down
    Lucidworks2026
  2. IL-0188 Rolling updates on EKS returned 502 errors for about 30 seconds because Kubernetes deleted old pods as soon as new pods passed readiness, before the ALB had marked the new pod targets healthy
    Yonghyun Kim2025
  3. IL-0020 During an EKS 1.30 upgrade the descheduler tried to reschedule every pod on the cordoned nodes at once and production workloads went down
    UK Ministry of Justice2025
  4. IL-0086 A deployment meant for a CircleCI test Kubernetes cluster also removed Traefik and Calico resources from the production cluster, taking down the CircleCI UI and runner.circleci.com
    CircleCI2021
  5. IL-0034 A crashing stats agent sidecar marked every pod unready and the service mesh pulled all services out of traffic
    Airbnb2019
  6. IL-0467 Boskos crash-looped after a node drain started it from a buggy StatefulSet image, failing Kubernetes PR jobs for 9 hours
    Kubernetes project2019

Breaking change in an upgrade12 cases

Component
Admission webhook4Ingress controller4Node operating system image4Cloud load balancer3ConfigMaps and Secrets2Client-side DNS resolver and cache2GKE control plane2Helm2kube-proxy2Calico1CDN and edge network1cert-manager1Cilium1CoreDNS1etcd1Istio1Karpenter1Kubernetes API server1kubelet1Linux kernel1Node pool1Persistent volumes and stateful sets1systemd-networkd1Web application firewall rules1
Contributing factor
Monitoring blind spot5Change with a delayed effect3Unrepresentative staging3Harmful manual action2No staged rollout2Ineffective rollback2Single point of failure2Deferred upgrades2Unverified assumption1Automation amplification1Untested backup1Configuration not in code1Unreviewed defaults1Cache-masked failure1Unpatched known bug1Limited debug access1No headroom1Partial rollout1Unpinned version1
Organisation
PortSwigger22i2c1Altinity1CircleCI1Datadog1Honeycomb1Jenkins project1Reddit1Sunwoo Park1UK Ministry of Justice1Wealthy1
  1. IL-0092 Upgrading ingress-nginx from 1.12.0 to 1.14.3 brought a newer ModSecurity and OWASP Core Rule Set whose false positives blocked HMPPS Auth, so users could not sign in to their applications
    UK Ministry of Justice2026
  2. IL-0405 An expired AWS Load Balancer Controller webhook certificate left no healthy targets and took BSEE SaaS down
    PortSwigger2026
  3. IL-0101 After Google forced a GKE control plane upgrade to 1.33.2 at a deprecation deadline, no new node could initialise, so autoscaling and spot node replacement stopped for 15 hours
    Wealthy2025
  4. IL-0079 A GKE node auto-upgrade to an ubuntu_containerd image with a kernel regression left kube-proxy unable to apply its ip6tables rules, so load balancer health checks failed and several clusters lost all ingress traffic
    Altinity2025
  5. IL-0404 All BSEE SaaS customer pods on EKS crash-looped after Karpenter moved nodes to Bottlerocket v1.26.0, which broke Java
    PortSwigger2024
  6. IL-0203 A routine Kubernetes cluster system update made the AWS Load Balancer Controller fail, so new pods were never registered with the ALB and Honeycomb's EU1 ingest and UI went down once the last old pods were replaced
    Honeycomb2024
  7. IL-0180 A GKE control plane auto-upgrade from 1.22 to 1.23 removed the v1beta1 Ingress and ValidatingWebhookConfiguration APIs that Istio 1.7.4 still used, and Istio stopped working in production
    Sunwoo Park2023
  8. IL-0060 Kubernetes 1.24 upgrade removed the master node label that Calico route reflectors selected on, dropping all pod routes and taking Reddit down for 314 minutes
    Reddit2023
  9. IL-0230 kube-proxy sync failures during a Kubernetes upgrade corrupted node iptables, and CircleCI jobs stopped starting
    CircleCI2023
  10. IL-0198 An automatic Ubuntu security update restarted systemd-networkd, which flushed the IP rules Cilium relied on and cut more than 60 percent of Datadog's Kubernetes nodes off the network across regions and cloud providers
    Datadog2023
  11. IL-0501 Every node of the UToronto Kubernetes cluster on Azure went NotReady and the JupyterHub became unreachable
    2i2c2022
  12. IL-0522 An ingress-nginx chart upgrade deleted the jenkins.io Ingress and Fastly served HTTP/503 errors for hours
    Jenkins project2021

Operator error7 cases

Component
Ingress controller4Database2Persistent volumes and stateful sets2Terraform2CI system1Cloud load balancer1Credential broker for pods1external-dns1Jobs and CronJobs1JupyterHub1kube-proxy1Model hub1Node pool1Prometheus1VPC route tables and firewall rules1
Contributing factor
No safeguard on destructive change6Unverified assumption1Unreviewed defaults1Harmful manual action1No runbook1No headroom1Overlapping failures1Deferred upgrades1Unsafe failure behaviour1
Organisation
UK Ministry of Justice22i2c1DevOps Hof1GitLab1OpenAI1UC Berkeley1
  1. IL-0095 An engineer deleted the prod namespace of the LEAP JupyterHub while believing the kubeconfig pointed at a new cluster, and the hub was down for about an hour
    2i2c2025
  2. IL-0244 A queued pull request destroyed the EKS subnet route table associations and Cloud Platform services went offline
    UK Ministry of Justice2024
  3. IL-0207 A routine task run from an engineer's local environment switched to production and deleted GitLab.com's Consul release, so Patroni lost Consul and GitLab.com returned 5xx errors to all traffic
    GitLab2022
  4. IL-0115 One of OpenAI's three production AKS clusters went offline through a human error, and rebuilding it took 10½ hours because an AKS Kubernetes upgrade broke the ingress and AKS cannot be downgraded
    OpenAI2022
  5. IL-0089 A test cluster deletion script run with the kube context pointing at the live-1 production cluster deleted its ingress controller, monitoring, logging, cert-manager, kiam and external-dns
    UK Ministry of Justice2020
  6. IL-0054 All ingress traffic lost while draining a GKE node pool because the load balancer only targeted new nodes without ingress-nginx pods
    DevOps Hof2019
  7. IL-0506 UC Berkeley DataHub logins failed and students got empty home directories after the JupyterHub database was overwritten
    UC Berkeley2017

Software defect11 cases

Component
Kubernetes API server3etcd2Ingress controller2kubelet2Persistent volumes and stateful sets2Cloud load balancer1Cluster autoscaler1Database1GKE control plane1GPU driver1JupyterHub1Linkerd1Linux kernel1Container memory limit1Node pool1Telemetry agent1Terraform1
Contributing factor
Monitoring blind spot3Single point of failure3Unverified assumption2Change with a delayed effect2Excessive blast radius1Unreviewed defaults1Ignored early signal1Unpatched known bug1Harmful manual action1No alert on limit1No configuration tests1No staged rollout1Overlapping failures1Ineffective rollback1Deferred upgrades1Unrepresentative staging1
Organisation
Binder Project2AirMap1Barush Mendez1DFDS1Linode1Monzo1Nordstrom1Palark1thredUP1ZenML1
  1. IL-0078 Enabling EKS secrets encryption with KMS through Terraform removed worker node IAM permissions and took down all ZenML Pro workspaces
    ZenML2025
  2. IL-0157 After a release, new nodes could not join Linode Kubernetes Engine clusters with IP ACL enabled, so autoscaling failed on about 20 to 25 clusters
    Linode2025
  3. IL-0138 A failed Nvidia driver build left broken kernel packages on the only Kubernetes master node, which hit a kernel panic on reboot after a power outage and took the homelab cluster down for six days
    Barush Mendez2025
  4. IL-0403 mybinder.org could not create pods for hours as the GKE gke-resource-quotas quota wrongly reported 15k pods in use
    Binder Project2022
  5. IL-0106 A malformed Traefik IngressRoute triggered a Traefik v2 bug once a node rollover restarted every Traefik pod, and all Kubernetes services returned 404
    DFDS2021
  6. IL-0196 Adding a second Redis replica and lowering memory limits in one RedisFailover change restarted the only Redis pod that held data, and Sentinel promoted the empty new replica, erasing the data
    Palark2020
  7. IL-0051 A third-party log shipping client leaked file descriptors with every new container until nodes failed with too many open files
    thredUP2018
  8. IL-0074 All AirMap platform services down as worker nodes went NotReady and pods crash-looped on network and DNS failures
    AirMap2018
  9. IL-0504 mybinder.org launches failed for about five hours as undeleted user pods piled up and filled the cluster
    Binder Project2018
  10. IL-0072 High etcd write latency made the API servers return stale data, and the production cluster flipped between two views of its state
    Nordstrom2017
  11. IL-0003 linkerd lost Kubernetes service discovery after an etcd reconfiguration, and restarting every linkerd instance turned failed payments into a full platform outage
    Monzo2017

unknown3 cases

Component
Calico1Container runtime1etcd1Kubernetes API server1kubelet1Multus meta-CNI1Node pool1Persistent volumes and stateful sets1
Contributing factor
Harmful manual action2Unverified assumption1Unreviewed defaults1Ignored early signal1Monitoring blind spot1Pages not delivered1Single point of failure1
Organisation
Binder Project1Fyra Labs1NRE Labs1
  1. IL-0087 After a cold reboot of RKE2 nodes, pods could not get IP addresses because Multus could not reach the Kubernetes API, and further reboots spread the failure to every node
    Fyra Labs2023
  2. IL-0059 Single-master Kubernetes control plane kept dying because the kubelet restarted a healthy etcd on a failing kubeadm liveness probe
    NRE Labs2018
  3. IL-0402 mybinder.org launches dropped to zero for hours because a JupyterHub pod on a lost GKE node kept its hub database PVC
    Binder Project2018

Certificate or credential expiry2 cases

Component
cert-manager1GKE control plane1Kubernetes API server1Node pool1RBAC and service accounts1Secrets store1
Contributing factor
Single point of failure2Unverified assumption1Limited debug access1Harmful manual action1Monitoring blind spot1
Organisation
GoCardless1Webhook Relay1
  1. IL-0113 The managed GKE control plane of Webhook Relay's main cluster failed because its API server certificates had never been rotated, and a manual credential rotation reported success without taking effect
    Webhook Relay2022
  2. IL-0460 Vault served an expired TLS certificate and almost every GoCardless service pod crash-looped, a 22-minute full outage
    GoCardless2020

DNS or network fault2 cases

Component
Container registry and its credentials2CoreDNS1Container memory limit1Object storage1
Contributing factor
Single point of failure2Harmful manual action1Overlapping failures1
Organisation
GitHub1Gitpod1
  1. IL-0517 Gitpod workspaces failed to start worldwide and lost data when CoreDNS lookups to 8.8.8.8 timed out in its clusters
    Gitpod2022
  2. IL-0199 A non-essential container exceeding its memory limit made GitHub's production pods unavailable, and their restarts failed because a DNS maintenance change had left the clusters unable to reach the image registry
    GitHub2020