Incident Library
Applied filters DomainKubernetesClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases259 cases

151 to 155 of 259

  1. IL-0139 Pods on GKE clusters with Calico network policy got stuck Terminating or Pending after upgrading to 1.22 or later because of a race condition in the Calico component
    Google Cloud2022
    Symptoms

    After upgrading to GKE 1.22 or later, clusters that use Calico network policy could have pods stuck in Terminating or Pending. Google Cloud posted the incident at 16:01 US/Pacific on 2022-09-15 as pods stuck terminating; from the 2022-09-21 update the description also named pods stuck Pending.

    Pods stuck in TerminatingPods Pending
    Trigger

    Upgrading a GKE cluster that uses Calico network policy to version 1.22 or later.

    Root cause
    Software defect
    Component
    CalicokubeletPod termination
    Contributing factor
    –
    Remediation
    • released a fix in GKE 1.24.4-gke.800 and made it available in 1.23 and 1.22
    • offered an internal patch through Google Cloud Support for affected customers
    • advised customers to restart the kubelet and calico-node to get stuck pods unstuck
    • advised customers without a strong need for Calico to disable Calico network policy
    Action items
    • Upgrade clusters on the Rapid, Regular and Stable release channels running 1.22 or 1.23 to the fixed version automatically
    Lessons

    A race in the Calico component can leave pods stuck Terminating, with FailedKillPod and KillPodSandbox errors that point at the network plugin, not at the workload. Restarting the kubelet and calico-node frees the stuck pods but does not remove the race; the fixed GKE version or Google's internal patch does. Clusters that use the cluster autoscaler are more likely to hit it.

    View case
  2. IL-0191 A label selector accidentally added to the Loki distributor Service matched only the new pods of a rollout, which took all write traffic, were OOM-killed and stalled the rollout
    Grafana Labs2022
    Symptoms

    During a rollout of the Grafana Loki distributor Deployment in one Grafana Cloud Logs environment, the write path went down for about 25 minutes. The newly started distributor pods were OOM-killed over and over, never passed their health and readiness checks, and the rollout stopped making progress.

    Service outageOOMKilledLoad imbalanceProbe failures
    Trigger

    Step 1 of moving the Loki ring off a separately deployed key-value store onto embedded memberlist added a new label, gossip_ring_member: 'true', to the distributor pods and Service, together with a small config change, and the change required a rollout of the distributors.

    Root cause
    Configuration change
    Component
    Configuration templatingService discoveryContainer memory limit
    Contributing factor
    Unreviewed defaultsIgnored early signalUnrepresentative staging
    Detection

    The first clue was kubectl get pods and the kube_deployment_status_replicas_updated metric showing that only 16 of 140 distributor pods had been updated.

    Remediation
    • scaled the distributor Deployment out horizontally so the load spread over more pods with the new label and the rollout could continue
    • reverted the migration step to end the outage
    • set service_ignored_labels:: [$._config.gossip_member_label] and passed it to k.util.serviceFor so the label is no longer copied into the Service selector
    Action items
    • Update the function that generates a Service from a Deployment or StatefulSet to handle edge cases, including this one
    Lessons

    A Service whose selector is built by copying every label of a workload changes which pods it matches as soon as a new label is added. During a rollout it then matches only the new pods, which receive all the traffic before there are enough of them. A few errors during a rollout in other environments hid the problem, because those environments gave distributors more memory and could absorb the shift. Manifest diffs kept for every pull request made it possible to see that the Service selector had changed.

    View case
  3. IL-0144 A systemd security update installed by unattended-upgrades lost the DNS configuration of Ubuntu 18.04 VMs, so AKS kubelets could not resolve registry names and pods failed with ImagePullBackOff
    Microsoft Azure2022
    Symptoms

    From 06:00 UTC on 30 August 2022, Azure VMs running Ubuntu 18.04 (bionic) with unattended-upgrades enabled began failing DNS resolution and losing network connectivity. Several Azure teams, among them the AKS and Azure Container Apps teams, were alerted by production alerts shortly after the update packages were published.

    DNS resolution failuresImage pull failuresMonitoring outage
    Trigger

    Unattended-upgrades on Ubuntu 18.04 VMs installed a Canonical security update published at 06:00 UTC on 30 August 2022, including systemd version 237-3ubuntu10.54.

    Root cause
    Breaking change in an upgrade
    Component
    Node operating system imagekubeletContainer registry and its credentialsClient-side DNS resolver and cache
    Contributing factor
    No staged rolloutExcessive blast radiusAutomation amplification
    Detection

    Several Azure teams, including AKS and Azure Container Apps, were alerted by production alerts shortly after the packages were published; incoming support cases then confirmed which Ubuntu versions were affected.

    Remediation
    • pointed customers to Canonical's description of the systemd bug and its fix
    • recommended rebooting affected VMs or running a template script that repairs DNS resolution
    • built automatic detection and remediation for AKS nodes and rolled it out across all regions
    • mitigated the remaining AKS customers manually through support
    Action items
    • Review the process that lets critical security updates bypass Safe Deployment Practices
    • Improve monitoring of the AKS data plane to alert on an uptick of errors
    • Review AKS monitoring algorithms so that similar VM errors are detected and alerted on
    • Establish an improved escalation path to Canonical during outages and add it to internal Technical Service Guides
    • AKS takes full control of the node security patch mechanism, with additional testing and controlled release of patches (estimated December 2022)
    • Work with Canonical to run dedicated tests on proposed packages before they are published for Azure IaaS VMs
    • Establish a process with Canonical to close the testing gap for the upgrade scenario (estimated March 2023)
    • Give AKS customers maintenance window control and environment staging for these patches (estimated December 2023)
    Lessons

    Security patches applied automatically by unattended-upgrades, once a day by default, and outside Azure's Safe Deployment Practices reached Ubuntu 18.04 VMs in every Azure region, so one systemd regression became a DNS outage across Azure. On AKS, lost node DNS showed up as image pull failures such as ImagePullBackoff, because kubelet could not resolve the registry. Azure plans to put node security patching under AKS's control with testing and controlled release; customers can use automatic node image upgrade for AKS, or automatic VM guest patching for VMs, instead of unattended-upgrades.

    View case
  4. IL-0501 Every node of the UToronto Kubernetes cluster on Azure went NotReady and the JupyterHub became unreachable
    2i2c2022
    Symptoms

    On 30 August 2022 at 11:15 AM Pacific time, the UToronto JupyterHub at jupyter.utoronto.ca stopped responding and users could not reach it at all. The outage was reported through 2i2c's support desk.

    Service outageNode NotReady
    Trigger

    Microsoft Azure rolled out an Ubuntu update to all Ubuntu 18.04 nodes, which triggered a systemd bug that broke networking.

    Root cause
    Breaking change in an upgrade
    Component
    Node operating system image
    Contributing factor
    Limited debug access
    Detection

    The outage was reported at 11:15 AM through 2i2c's Freshdesk support desk because jupyter.utoronto.ca was not responding. At 11:25 AM the customer's representative forwarded the Microsoft advisory on the same ticket, which pointed the team to the cause.

    Remediation
    • Started an upgrade of the cluster's Kubernetes and node versions, which replaced all old nodes with new working ones
    Action items
    • Document how to SSH into the Kubernetes nodes
    • Find a way to be notified of Azure advisories
    Lessons

    Once the cause was known, a single action brought the hub back quickly. The team had no documented way to SSH into its Kubernetes nodes, and the customer's representative saw the Azure advisory and pointed it out on the support ticket, which saved a lot of time.

    View case
  5. IL-0107 Kyverno stuck in a crash loop behind a webhook with failurePolicy fail blocked every change to Kubernetes Service objects for 88 minutes
    DFDS2022
    Symptoms

    At 11:38 UTC on 3 August 2022 an end-user team told the platform team that applying a Service resource to one of the Kubernetes clusters failed.

    Deploys failingContainer crash loop
    Trigger

    A certificate chain in a third-party application expired, through no action of DFDS, and the outdated Kyverno installation got stuck in a crash loop.

    Root cause
    Certificate or credential expiry
    Component
    Policy engineAdmission webhook
    Contributing factor
    Deferred upgradesUnreviewed defaultsMonitoring blind spotIgnored early signal
    Detection

    An end-user team alerted the platform team at 11:38 UTC that applying a Service resource failed; the platform team started looking into it at 11:50 UTC.

    Remediation
    • reproduced the issue in a sandbox cluster and deployed the fix there first
    • updated Kyverno to a newer Helm chart version in the affected production cluster
    Action items
    • Implement monitoring on Kyverno
    • Create internal documentation on the Kyverno setup
    • Investigate the exact certificate issue in the third-party application
    • Consider changing the failurePolicy on ClusterPolicy resources going forward
    Lessons

    Operational issues should be added to the 'Ops Day to Day' board from the start.

    View case
Check your cluster for this

These 259 cases involve Node pool and Ingress controller. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause14

Configuration change46 cases

Component
Cloud load balancer9Node pool8Kubernetes API server7Configuration templating5Terraform5Ingress controller4AWS VPC CNI4etcd3kubelet3Node launch template3Container memory limit3Telemetry agent3Admission webhook2Argo CD2cert-manager2Cilium2CNI plugin2conntrack table2Container runtime2CoreDNS2Client-side DNS resolver and cache2GKE control plane2HAProxy2Helm2Linux kernel2Persistent volumes and stateful sets2Pod priority and preemption2In-memory cache1CDN and edge network1CI system1Cluster dashboard1Configuration management tool1ConfigMaps and Secrets1Coordination and key-value store1Credential broker for pods1Database1Database proxy and connection pooler1Authoritative DNS servers and records1external-dns1Horizontal pod autoscaler1HTTP and RPC client library1Jobs and CronJobs1Karpenter1Application logging library1Node root disk1Node operating system image1Node iptables rules1Pod termination1Prometheus1RBAC and service accounts1Container registry and its credentials1Service discovery1
Contributing factor
Monitoring blind spot13Unverified assumption12Excessive blast radius11Change with a delayed effect10Automation amplification8Unreviewed defaults8No configuration tests8Retry storm6Missed in review6No safeguard on destructive change5No staged rollout5Unrepresentative staging5No runbook4Ignored early signal3Limited debug access3No alert on limit3Unknown limit2Harmful manual action2No health probes2No headroom2Signal buried in noise1Untested backup1Configuration not in code1Cache-masked failure1No resource requests or limits1Overlapping failures1Partial rollout1Shared credential1Single point of failure1Unpinned version1
Organisation
DFDS5OpenAI4Grafana Labs3UK Ministry of Justice3DigitalOcean2GitLab2Wikimedia Foundation2Adevinta1AdGuard VPN1Adyen1Aqua Security1CoW Protocol1Datadog1Discord1GitHub1Google Cloud1Hans H. Kim1InfluxData1Jenkins project1Jetstack1JW Player1LIFULL1LINE Plus1MIT Open Learning1MUSINSA1NU.nl1Pusher1Python Software Foundation1Skyscanner1Spotify1Stytch1Target1
  1. IL-0179 With DEBUG logging and synchronous log output, a high-traffic service filled the 64 KB stdout pipe, so request threads blocked in write() while holding database connections and HikariCP ran dry, failing requests in 30-second bursts
    MUSINSA2026
  2. IL-0246 New pods on Adyen's big data Kubernetes nodes could not start, failing with "Cilium API client timeout exceeded"
    Adyen2026
  3. IL-0512 xpro.mit.edu returned 503 for an hour when its Kubernetes/APISIX origin certificate expired after renewal failed
    MIT Open Learning2026
  4. IL-0499 Discord voice and video calls hung at Awaiting Endpoint for three hours after 17% of sessions stopped at once
    Discord2026
  5. IL-0200 Newly provisioned Kubernetes nodes ran an incompatible etcd version and broke cluster consensus in several GitHub production clusters, causing 5XX errors on Git operations and page loads
    GitHub2026
  6. IL-0063 VPN outage after recreating a Redis StatefulSet restarted the whole Redis cluster without readiness checks
    AdGuard VPN2025
  7. IL-0122 An invalid value in a configuration shared by several ChatGPT services put pods into crash loops and raised error rates
    OpenAI2025
  8. IL-0192 A Crossplane composition merge-policy change for a TLS policy update switched Kubernetes Services from LoadBalancer to ClusterIP and deleted the load balancers behind about 25% of Grafana Cloud services
    Grafana Labs2025
  9. IL-0005 A new telemetry service overloaded the Kubernetes API servers of large clusters and broke DNS-based service discovery for all OpenAI services
    OpenAI2024
  10. IL-0118 A global change to Kubernetes namespace labels made the networking layer recompute metadata and overwhelmed the control planes of three of OpenAI's largest GPU clusters
    OpenAI2024
  11. IL-0130 Deployments of large models from Vertex Model Garden failed with "Model server never became ready" in most regions for 13 days after a storage provisioning configuration change
    Google Cloud2024
  12. IL-0114 An unpinned AWS VPC CNI plugin was updated automatically during a capacity expansion, leaving new pods Pending without IP addresses and stopping CoW Protocol settlements for about 3.5 hours
    CoW Protocol2024
  13. IL-0152 An incorrect configuration applied during maintenance made an internal infrastructure cluster automatically delete the control planes of 200 DOKS clusters in FRA1
    DigitalOcean2024
  14. IL-0153 DOKS worker nodes went NotReady at 06:00 UTC on three days because a systemd timer in the Debian image ran apt upgrade and killed it mid kernel upgrade, breaking node networking
    DigitalOcean2023
  15. IL-0019 An EKS Terraform module upgrade shrank node root disks from 100 GB to 20 GB and image pulls failed with no space left on device
    UK Ministry of Justice2023
  16. IL-0459 MediaWiki on Kubernetes terminated TLS with a wrong certificate after a deployment, and 1% of requests failed
    Wikimedia Foundation2023
  17. IL-0017 AWS VPC CNI stopped allocating pod IP addresses on six nodes of the live cluster, leaving new pods in ContainerCreating
    UK Ministry of Justice2023
  18. IL-0473 Jenkins infrastructure went fully down after a pod IP range fix deleted every AKS node pool and could not recreate them
    Jenkins project2023
  19. IL-0091 Changing the worker node instance type through Terraform recycled all nodes at once and deleted users' production pods together
    UK Ministry of Justice2023
  20. IL-0068 Anonymous access to the Kubernetes API server used to plant a ClusterRoleBinding backdoor and a cryptominer DaemonSet
    Aqua Security2023
  21. IL-0186 Pods on one Kubernetes worker node could not reach external HTTP and HTTPS because storage components had added nat PREROUTING rules that redirected ports 80 and 443 to local ports
    Hans H. Kim2023
  22. IL-0231 Deleting EKS managed node groups also deleted Karpenter's instance profile, and every Stytch worker node shut down
    Stytch2023
  23. IL-0109 A retried node rollover on a DFDS EKS cluster set the Auto Scaling group back to 15 instances, and the default termination policy removed every healthy worker node, leaving the cluster with no nodes
    DFDS2023
  24. IL-0116 A CNI configuration change incompatible with a hardware variant that exists only in production cut network connectivity in an OpenAI GPU cluster and took text-davinci-003 fully down
    OpenAI2023
  25. IL-0513 PyPI backends on Kubernetes were unreachable over HTTPS for 20 minutes when the Ingress ELB lost its HTTPS listener
    Python Software Foundation2022
  26. IL-0177 WebSocket connections to LINE's messaging-hub on Kubernetes dropped all at once whenever another service's change reloaded a shared HAProxy L7 load balancer, and the reconnect wave overloaded a token API and filled nf_conntrack on Nginx
    LINE Plus2022
  27. IL-0191 A label selector accidentally added to the Loki distributor Service matched only the new pods of a rollout, which took all write traffic, were OOM-killed and stalled the rollout
    Grafana Labs2022
  28. IL-0009 A duplicated generated name made Argo CD replace the InfluxDB Cloud production Application and delete the whole production environment
    InfluxData2022
  29. IL-0461 Every rollout of Datadog's metrics query service on Kubernetes raised errors that the logs blamed on DNS
    Datadog2022
  30. IL-0208 A GitLab chart bump that renamed Kubernetes Service and Deployment ports made the GCP load balancer reconfigure, and GitLab.com slowed down and returned intermittent 50x errors
    GitLab2022
  31. IL-0442 Older pods on LIFULL's KEEL Kubernetes cluster were preempted all at once and the external error rate spiked
    LIFULL2021
  32. IL-0111 Switching the main Kubernetes ingress controller from Traefik v1 to v2 stopped X-Forwarded headers being passed, which broke the Okta login flow for every service using Okta and stopped DFDS online bookings for over an hour
    DFDS2021
  33. IL-0010 A templating change that dropped its {{ }} delimiters made Argo CD delete all 478 Skyscanner services in every region
    Skyscanner2021
  34. IL-0209 A too-high HPA CPU target kept GitLab.com web pods from scaling up as traffic moved from VMs to Kubernetes, breaching the web latency SLO for 15 minutes
    GitLab2021
  35. IL-0105 Replacing the AWS VPC CNI during a Terraform refactoring detached the ALB target groups, and every service in the Kubernetes cluster returned 503
    DFDS2021
  36. IL-0108 An Alcide admission controller applied to the production cluster during a workshop made create and update requests on Deployments time out
    DFDS2020
  37. IL-0365 A Terraform change to auto-scaling groups detached all worker nodes from both ALBs and cut off every cluster service
    DFDS2020
  38. IL-0189 A JVM service moved from EC2 to Kubernetes answered up to ten times slower because KIAM's 15-minute credentials made the AWS Java SDK fetch new credentials on every request
    Adevinta2019
  39. IL-0458 Toolforge Kubernetes API server failed to start after a restart, logging "etcd cluster is unavailable or misconfigured"
    Wikimedia Foundation2019
  40. IL-0008 An unresponsive OPA validating webhook crash-looped the API server during a GKE master upgrade, and node auto-repair then recreated every node
    Jetstack2019
  41. IL-0004 A new default pod priority let a new Cortex cluster preempt every production ingester in a cascade, taking Grafana Cloud Hosted Prometheus down
    Grafana Labs2019
  42. IL-0061 A Terraform pull request merged out of order deleted a production GKE cluster, and recreating it deleted a second one
    Spotify2019
  43. IL-0066 Broken ConfigMap change lay dormant for weeks, then took down Pusher's edge proxy when a cluster upgrade replaced its pods
    Pusher2019
  44. IL-0052 A cryptocurrency miner ran on every node of the development and staging clusters through a public, unauthenticated Weave Scope dashboard
    JW Player2019
  45. IL-0035 Worker nodes went NotReady after a new ElastAlert rule without memory limits used up node memory
    NU.nl2019
  46. IL-0055 A failing CronJob with the default concurrency policy piled up thousands of restarting pods and took down a cluster
    Target2018

Capacity exhaustion69 cases

Component
Container memory limit15Node pool13CoreDNS10Database10Kubernetes API server10Prometheus8Client-side DNS resolver and cache7kubelet7Telemetry agent6In-memory cache5etcd5Pod scheduling and placement5Horizontal pod autoscaler4Ingress controller4Linux kernel4Persistent volumes and stateful sets4Cloud block storage3CI system3conntrack table3Container CPU limit3Helm3Jobs and CronJobs3kube-proxy3Node root disk3VPC subnet or pod address range3Cluster autoscaler2CNI plugin2Container runtime2JupyterHub2Karpenter2kube-dns2Managed Kubernetes control plane and provider API2Container registry and its credentials2AWS VPC CNI2VPC DNS resolver2Admission webhook1BGP routing1Calico1Cilium1Cloud controller manager1Credential broker for pods1Elasticsearch or OpenSearch1GKE control plane1HAProxy1Liveness and readiness probes1Model inference server1InfiniBand fabric1Istio1Language runtime settings1Model hub1NAT gateway1Network control plane (SDN)1NIC, cable and switch port link1Cluster authentication provider1Pod termination1Service discovery1
Contributing factor
Unreviewed defaults23Monitoring blind spot16Unknown limit15Single point of failure13Excessive blast radius10Harmful manual action10Retry storm10No alert on limit9Unverified assumption8Automation amplification8No resource requests or limits8No capacity planning6No headroom5Ignored early signal4Unrepresentative staging4No caching3Ineffective rollback3Change during peak hours2Change with a delayed effect2Limited debug access2No runbook2No staged rollout2Overlapping failures1Partial rollout1Health check tied to a dependency1
Organisation
UK Ministry of Justice52i2c4Buildkite3IONOS Cloud3Zalando3Backpackr2loveholidays2Palark2Scaleway2Adevinta1Airbnb1BC Government1Binder Project1Blue Matador1Chatwork1Civis Analytics1CyberAgent1Cybozu1Datadog1DFDS1DigitalOcean1freedesktop.org1Fullstaq1GitHub1GitLab1Google Cloud1Honeycomb1incident.io1Kevin1Kubernetes project1LINE1Linode1Magic Pages1Mercari1Microsoft1MonotaRO1Moonlight1MUSINSA1Neon1Onkar Raghunath Shelke1PostHog1RebelMouse1Render1Saifeddine Rajhi1Saltside1Temporal1Tinder1UC Berkeley1Universe1unknown1Wikimedia Foundation1이언철1
  1. IL-0071 Worker node running out of inodes while disk space looked fine, because containerd kept a full node_modules tree in each of several image layer snapshots
    Onkar Raghunath Shelke2026
  2. IL-0102 CoreDNS pods on a production EKS cluster ran out of memory under a burst of new pods, endpoints and nodes, taking down the Buildkite web UI, APIs and job dispatch
    Buildkite2026
  3. IL-0471 GitHub Actions runs failed or sat queued for 9 hours after a routine deployment's pod replacement saturated capacity
    GitHub2026
  4. IL-0155 Some DOKS clusters in NYC1 saw intermittent DNS timeouts and NodeNotReady events because CoreDNS ran on shared-CPU Droplet worker nodes
    DigitalOcean2026
  5. IL-0470 Buildkite notifications were delayed after Prometheus storage filled and EKS notification workers could not autoscale
    Buildkite2026
  6. IL-0548 Buildkite's EKS Prometheus server was killed for memory again and again, so services stopped scaling and jobs lagged
    Buildkite2026
  7. IL-0435 Error rates rose as Aurora MySQL killed connections and declined queries to avoid OOM while load metrics stayed flat
    CyberAgent2026
  8. IL-0464 IONOS Managed Kubernetes control planes went unreachable for minutes at a time as their shared etcd grew and fragmented
    IONOS Cloud2026
  9. IL-0465 IONOS Container Registry image pushes and pulls failed with 504 Gateway Timeout
    IONOS Cloud2026
  10. IL-0350 Hosts in an IONOS TXL cluster lost connectivity when an InfiniBand fabric resweep failed on a full multicast table
    IONOS Cloud2026
  11. IL-0181 A traffic spike from an external community post pushed product API pods into CPU throttling under a 1-core limit, and an e-commerce product page slowed down while every pod stayed Running
    Kevin2026
  12. IL-0581 29CM product display servers lost pods and saturated Netty event loops on the first day of the Igu Week sale
    MUSINSA2025
  13. IL-0100 PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queries
    PostHog2025
  14. IL-0413 LLM inference engines timed out or ran out of resources under load in about 60% of 156 severe incidents at Microsoft
    Microsoft2025
  15. IL-0048 DNS lookups timed out in a large EKS cluster when CoreDNS hit the 1024 packets-per-second link-local limit, driven by sudo hostname lookups
    unknown2025
  16. IL-0067 Redis filled its persistent volume, corrupted its append-only file and crash-looped on every restart
    BC Government2025
  17. IL-0080 kube-dns on Mercari's GKE clusters could not keep up with peak-hour DNS query rates, and a Sev1 production outage followed when several services failed DNS resolution
    Mercari2025
  18. IL-0082 Losing one CoreDNS pod during a planned EKS node reroll overloaded the remaining CoreDNS pods, causing DNS timeouts and errors across multiple services at RebelMouse
    RebelMouse2025
  19. IL-0006 VPC subnets ran out of IP addresses because the AWS VPC CNI kept IPs on full nodes, and WARM_IP_TARGET changes then blocked pod starts after every pod deletion
    Neon2025
  20. IL-0099 User servers on 2i2c's Earthscope hub took over 15 minutes to start and kernels were killed during a workshop, because a profile option guaranteed 8 GB of memory with no limit on 32 GB nodes
    2i2c2025
  21. IL-0170 During a Lunar New Year event, traffic more than three times normal filled the nf_conntrack table on the EKS node running the Istio ingress gateway, dropping packets and taking down Backpackr's web and app services
    Backpackr2025
  22. IL-0027 All CoreDNS pods landed on one node, the node ran out of memory, and DNS for the whole service went down
    Backpackr2025
  23. IL-0085 A control plane node of Magic Pages' Kubernetes cluster on Hetzner Cloud ran out of memory, the cluster's own calls then used up the Hetzner API rate limit, and all customer websites were down for about 7.5 hours
    Magic Pages2025
  24. IL-0204 An etcd memory spike overloaded the control plane of a Render Frankfurt cluster, and DNS then failed because CoreDNS ran on the rebooted control plane nodes and could not start without the API server
    Render2024
  25. IL-0007 The AWS VPC CNI could not assign IP addresses to pods as an EKS cluster ran out of IPv4 space
    Adevinta2024
  26. IL-0232 Calico pods in the codfw WikiKube cluster hit their resource limits and crash-looped, taking nodes out one by one
    Wikimedia Foundation2024
  27. IL-0440 Pods on Cybozu's Neco Kubernetes platform got frequent no such host and i/o timeout DNS errors from dropped packets
    Cybozu2024
  28. IL-0042 Application logs stopped reaching Elasticsearch and OpenSearch for hours and were lost when OpenSearch ran out of disk space
    UK Ministry of Justice2024
  29. IL-0044 Prometheus OOMKilled in a loop while replaying its write-ahead log after a kube-prometheus-stack upgrade
    Saifeddine Rajhi2024
  30. IL-0182 Too few CoreDNS replicas on EKS were overloaded during traffic spikes, raising latency and cutting the network for task pods, and the over-provisioned, tightly spread CoreDNS pods added later blocked Karpenter consolidation
    이언철2024
  31. IL-0046 Prometheus restarted again and again because its WAL replay took longer than the startup probe allowed
    UK Ministry of Justice2024
  32. IL-0110 A Grafana Agent pod's high memory use and crash-looping Hangfire pods drove a Kubernetes node into system OOM and NotReady, twice in one day
    DFDS2024
  33. IL-0168 Simultaneous cluster auto-upgrades overloaded Scaleway's Kubernetes API, and a node authentication retry loop kept Kapsule and Kosmos nodes NotReady in FR-PAR
    Scaleway2024
  34. IL-0169 The Scaleway Kubernetes API in FR-PAR ran out of memory under simultaneous deployments, and because it also handled node authentication, Kapsule and Kosmos nodes went NotReady
    Scaleway2024
  35. IL-0202 A burst of queries and database writes exhausted connection pools and memory on Honeycomb's ingest hosts, and Kubernetes crash-loop back-off kept failed hosts offline so ingest capacity stayed too low to recover
    Honeycomb2023
  36. IL-0386 After a move to Google Cloud, Postgres connection timeouts and memcached i/o timeouts spiked on GKE nodes
    incident.io2023
  37. IL-0433 Shifting API traffic from EC2 to EKS on Fargate overloaded two CoreDNS pods and caused mass name-resolution errors
    MonotaRO2023
  38. IL-0043 Application logs missing in Kibana after a newer Fluent Bit version changed its chunk drop strategy
    UK Ministry of Justice2023
  39. IL-0016 Prometheus on the live cluster ran out of memory and was OOMKilled, and the monitoring node it ran on went NotReady
    UK Ministry of Justice2023
  40. IL-0028 Temporal History service latency rose because its pods were CPU-throttled while CPU usage read below 60 percent
    Temporal2023
  41. IL-0195 The kube-apiservers of a cluster with more than 200 nodes ran out of memory and crashed one after another under mass LIST requests from Cilium agents
    Palark2023
  42. IL-0165 Some LKE users in Frankfurt could not reach their kube-apiservers with kubectl after backend services failed under memory constraints
    Linode2023
  43. IL-0194 After a kube-apiserver restart, pod LIST requests from every pod of a Vector DaemonSet nearly tripled etcd memory and restarted all master nodes
    Palark2023
  44. IL-0070 Critical pods on core nodes evicted when Prometheus began scraping about 200 nodes after a dask-gateway scale-up
    2i2c2023
  45. IL-0500 UToronto JupyterHub logins and server starts returned 500 errors when the hub's sqlite database disk filled up
    2i2c2022
  46. IL-0140 The GKE API in us-central1 returned errors and high latency for 3 hours 19 minutes after an unexpected rise in backend traffic that client retries made worse
    Google Cloud2022
  47. IL-0206 Consul agents in GitLab.com's GKE pods were repeatedly OOM-killed, so the Rails pods lost database replica discovery and sent every query to the primary
    GitLab2022
  48. IL-0445 Chatwork's service on EKS became unstable as conntrack tables on some nodes overflowed and connections were dropped
    Chatwork2021
  49. IL-0094 Student pods filled the shared nodes of a JupyterHub cluster and starved the hub's core pods, freezing the hub
    2i2c2020
  50. IL-0093 Ingress controller pods crash-looped one after another because the leader pod had exhausted its shared memory
    UK Ministry of Justice2020
  51. IL-0176 Prometheus in a 235-node Kubernetes cluster crash-looped because over two million series exhausted host memory and the kernel OOM killer chose it first, as Burstable QoS gave it oom_score_adj 999
    LINE2020
  52. IL-0013 The conntrack table filled up on downsized GKE nodes and a high-traffic aggregation proxy saw climbing connection resets
    loveholidays2020
  53. IL-0015 A GKE cluster's /16 pod address range ran out at 256 nodes, leaving new pods Pending and node scale-ups failing
    loveholidays2020
  54. IL-0033 Uneven load across kube-dns pods and a DNS packets-per-second limit pushed service latency to 20 seconds
    Airbnb2019
  55. IL-0026 Kubernetes nodes hit SystemOOM because fluentd-sumologic log shipper pods had no memory limits
    Blue Matador2019
  56. IL-0002 CI builder pods could not get AWS IAM credentials because kubelet fell minutes behind its API rate limit
    Zalando2019
  57. IL-0076 Kubernetes masters went down after Dex filled etcd with AuthRequest objects and deleting its namespace started an endless list-and-timeout loop
    Fullstaq2019
  58. IL-0401 Prow jobs on the k8s-prow-build GKE cluster failed and merges stalled as nodes hit IOPS throttling and kernel panics
    Kubernetes project2019
  59. IL-0037 Image registry rate-limited a whole region after a crash-looping DaemonSet with imagePullPolicy Always pulled through three NAT instances
    Datadog2019
  60. IL-0064 CPU-heavy pods packed onto one Google Cloud node drove it to kernel panics, and the crash moved from node to node until the site was down
    Moonlight2019
  61. IL-0075 Kubernetes API servers pegged on CPU when the cluster grew to about 290 nodes, traced to Datadog agents polling the API
    Civis Analytics2019
  62. IL-0472 Node-to-pod traffic and DNS failed on Tinder's 605-node Flannel cluster when every node's ARP cache passed gc_thresh3
    Tinder2019
  63. IL-0001 Cluster-wide DNS outage after CoreDNS pods were OOM-killed again and again under a burst of DNS queries
    Zalando2019
  64. IL-0031 Kubernetes API server OOM-killed, leaving the Skipper ingress without routes; the talk points to a misconfigured CronJob manifest
    Zalando2018
  65. IL-0518 gitlab.freedesktop.org on GKE ran short of volume space and a planned 30-minute storage upgrade became a 4-hour outage
    freedesktop.org2018
  66. IL-0503 mybinder.org launches failed after a GKE node ran out of memory and its nginx-ingress proxy pod crash-looped
    Binder Project2017
  67. IL-0036 A reindex Job that kept failing exhausted one node after another until the whole cluster was down
    Universe2017
  68. IL-0040 Parallel Helm installs saturated small gp2 root volumes on a kops cluster and left all three workers NotReady with the kubelet or Docker down
    Saltside2017
  69. IL-0468 New JupyterHub student pods stuck Pending on MaxVolumeCount after a memory limit cut with node size unchanged
    UC Berkeley2017

Dependency failure9 cases

Component
Node operating system image3Database2Node pool2Admission webhook1Cloud block storage1Container runtime1Credential broker for pods1Data center power and cooling1Authoritative DNS servers and records1etcd1GPU driver1Liveness and readiness probes1InfiniBand fabric1Istio1Persistent volumes and stateful sets1Container registry and its credentials1Service discovery1Telemetry agent1VPC DNS resolver1
Contributing factor
Automation amplification3Monitoring blind spot3Single point of failure3Excessive blast radius2Unreviewed defaults2Unknown limit2Unpatched known bug1Harmful manual action1No alert on limit1No runbook1Overlapping failures1
Organisation
Linode2FREE NOW1Google Cloud1Magic Pages1Nebius1Target1Tetrate (unnamed U.S. federal customer)1Yonghyun Kim1
  1. IL-0133 A storm shut down cooling at Nebius's us-central1 data center, and a disk hot-plug bug kept Managed Kubernetes control planes down for hours after power returned
    Nebius2026
  2. IL-0187 A brief Spring Cloud Config Server downtime failed every client application's liveness probe, because the actuator health check included the config server, and Kubernetes restarted the pods in a kill loop
    Yonghyun Kim2025
  3. IL-0162 LKE users could not reach their cluster control planes or the LKE Dashboard because an earlier database update made the internal LKE DNS service hang
    Linode2025
  4. IL-0103 During an EKS node AMI update, Istio sidecars crash-looped because hundreds of pods restarting at once overloaded the external certificate authority behind istiod
    Tetrate (unnamed U.S. federal customer)2025
  5. IL-0159 Docker Hub rate-limited Linode's IPv6 address space with 429 Too Many Requests, breaking image pulls for LKE and LKE-E clusters
    Linode2025
  6. IL-0489 Rejoining a failed node of Magic Pages' three-node MySQL cluster broke its metadata and took customer websites down
    Magic Pages2025
  7. IL-0021 GKE GPU nodes on Container-Optimized OS could not download NVIDIA drivers (403 Forbidden) and came up without working GPUs
    Google Cloud2024
  8. IL-0039 New Kubernetes worker nodes could not join any cluster because a package URL hardcoded in kops nodeup broke
    FREE NOW2019
  9. IL-0073 Intermittent Kafka connectivity overloaded Docker daemons, set off a node rescheduling cascade and poisoned the Consul gossip mesh
    Target2019

DNS or network fault10 cases

Component
conntrack table4CoreDNS3kube-proxy3Client-side DNS resolver and cache2Linux kernel2Container registry and its credentials2AWS VPC CNI2Cloud load balancer1Horizontal pod autoscaler1Ingress controller1Kubernetes API server1kubelet1Container memory limit1Node operating system image1Object storage1Pod termination1Telemetry agent1
Contributing factor
Unreviewed defaults6Unverified assumption2Ignored early signal2Unknown limit2Single point of failure2Automation amplification1Unpatched known bug1Harmful manual action1No caching1Overlapping failures1
Organisation
MindTickle2Adevinta1DigitalOcean1GitHub1Gitpod1Lotte E-commerce1Preply1Prezi1Uzabase1
  1. IL-0448 Requests from Uzabase's Kubernetes Job pods to an LLM agent randomly never got responses when they ran past 15 minutes
    Uzabase2026
  2. IL-0154 Line cards on an SFO3 core router rebooted together and halved backbone capacity, and DOKS nodes whose guest networking did not recover stayed NotReady with degraded API servers until all affected nodes in the SFO3 DOKS fleet were recycled
    DigitalOcean2024
  3. IL-0083 Intermittent 5xx errors in a multi-tenant Kubernetes cluster came from internal DNS: the node-local dnsmasq hit its 150 concurrent query limit, did not cache cluster.local answers, and ndots search lookups flooded CoreDNS
    Adevinta2023
  4. IL-0178 On EKS, HPA scale-in returned 502 errors because containers stopped before their pod IPs left the Service endpoints, and after blue/green switches conntrack kept sending established connections to deleted pods behind an instance-mode ALB
    Lotte E-commerce2022
  5. IL-0517 Gitpod workspaces failed to start worldwide and lost data when CoreDNS lookups to 8.8.8.8 timed out in its clusters
    Gitpod2022
  6. IL-0199 A non-essential container exceeding its memory limit made GitHub's production pods unavailable, and their restarts failed because a DNS maintenance change had left the clusters unable to reach the image registry
    GitHub2020
  7. IL-0056 Calls from EKS pods to services behind an NLB in another VPC failed intermittently because the AWS VPC CNI did source NAT on the node
    MindTickle2020
  8. IL-0049 Partial DNS outage after CoreDNS scaled down and kube-proxy failed to delete stale conntrack entries
    Preply2020
  9. IL-0057 Higher p99 latency after moving a reverse proxy to EKS, from outgoing connections delayed by a SNAT conntrack race
    Prezi2019
  10. IL-0047 Pod-to-pod requests delayed by 5, 10 or 15 seconds because of DNS lookups lost to a conntrack race
    MindTickle2019

Cloud provider outage15 cases

Component
Cloud load balancer4Cluster autoscaler4Database4GKE control plane4Ingress controller3Kubernetes API server3Node pool3AKS control plane2Horizontal pod autoscaler2CDN and edge network1Cloud controller manager1Coordination and key-value store1Helm1kubelet1Node root disk1Persistent volumes and stateful sets1Secrets store1Web application firewall rules1
Contributing factor
Overlapping failures5Limited debug access3Single point of failure3Unverified assumption2Excessive blast radius2No headroom2Automation amplification1Ignored early signal1Unpatched known bug1Harmful manual action1Monitoring blind spot1No capacity planning1Pages not delivered1Health check tied to a dependency1Retry storm1Unrepresentative staging1
Organisation
2i2c1ANDPAD1Binder Project1Devsisters1GitLab1Grafana Labs1Harness1loveholidays1Microsoft Azure1Namespace1OpenAI1Python Software Foundation1Twingate1UC Berkeley1UK Ministry of Justice1
  1. IL-0438 Builder Work became almost entirely unreachable with 504 errors when the Amazon CloudFront VPC Origins feature failed
    ANDPAD2026
  2. IL-0463 Secure Connect connectors timed out after a GKE control plane upgrade left Harness's internal load balancer unreachable
    Harness2026
  3. IL-0542 Twingate authentication and authorization requests failed at higher rates after traffic moved onto a smaller GKE cluster
    Twingate2025
  4. IL-0119 ChatGPT web service pods crash-looped on failing health checks after a Cosmos DB failure, leaving too few healthy pods to serve the web layer
    OpenAI2025
  5. IL-0084 Packet drops on a bare metal provider's vSwitch forced Namespace to rebuild its fra3 region and left global capacity short, and the next day slow NVMe writes on a control plane node drove the Kubernetes API server to very high CPU
    Namespace2024
  6. IL-0145 A power sag and chiller failure in one Australia East availability zone took down the AKS control plane's compute and its non-zone-redundant SQL Database operation queue, blocking AKS create, update and delete operations in Australia East and Australia Southeast
    Microsoft Azure2023
  7. IL-0096 A GKE node pool pinned to the europe-west2-b zone could not scale for 6.5 hours because Google ran out of resources in that zone (ZONE_RESOURCE_POOL_EXHAUSTED)
    2i2c2022
  8. IL-0514 PyPI's backends stopped responding for 7 minutes as rescheduled pods could not start while Consul and Vault were down
    Python Software Foundation2022
  9. IL-0364 A quarter of Cloud Platform ingress traffic was slow because an AWS NLB stopped health-checking targets in one zone
    UK Ministry of Justice2022
  10. IL-0205 The GKE cluster autoscaler could not add nodes to GitLab.com's production cluster because Google Cloud ran out of SSD persistent disks in us-east1
    GitLab2021
  11. IL-0172 An AWS Tokyo data centre cooling failure took down 6 of 60 CockroachDB nodes on Kubernetes within 22 minutes, and ranges lost quorum because the Helm-deployed cluster had no availability zone locality configured
    Devsisters2021
  12. IL-0193 A GCP persistent disk degradation took down a Cassandra StatefulSet whose OrderedReady policy blocked recovery, and Metrictank could not reconnect to the recreated pods, causing a 23-hour Graphite query outage
    Grafana Labs2019
  13. IL-0014 A GKE upgrade to a withdrawn version stalled on slow drains and a stuck node pool, then broke an ingress and confused the HPA
    loveholidays2019
  14. IL-0469 The managed Azure Kubernetes API server for the data100 hub stopped answering, so the stopped hub pod was not recreated
    UC Berkeley2019
  15. IL-0505 kubectl logs failed on GKE with "No SSH tunnels currently open" and mybinder.org build logs stopped reaching users
    Binder Project2019

Software defect46 cases

Component
Ingress controller7Linux kernel7Node operating system image7kubelet6Kubernetes API server5Database4etcd4Node pool4Telemetry agent4Cloud load balancer3Container CPU limit3Admission webhook2Argo CD2BGP routing2GPU driver2Network control plane (SDN)2Persistent volumes and stateful sets2Pod termination2Terraform2Azure CNI1Calico1cert-manager1Cluster autoscaler1Configuration templating1conntrack table1CoreDNS1Credential broker for pods1Client-side DNS resolver and cache1Authoritative DNS servers and records1Elasticsearch or OpenSearch1GKE control plane1HAProxy1Liveness and readiness probes1Horizontal pod autoscaler1Istio1JupyterHub1Karpenter1Kubernetes controller manager1kube-proxy1Language runtime settings1Node launch template1Linkerd1Managed Kubernetes control plane and provider API1Container memory limit1Network switch1Pod scheduling and placement1Policy engine1Serverless function platform1AWS VPC CNI1
Contributing factor
Unverified assumption8Monitoring blind spot8Unreviewed defaults7Automation amplification6Unpatched known bug6Excessive blast radius5Change with a delayed effect3Ignored early signal3Single point of failure3Unknown limit2Limited debug access2Harmful manual action2No alert on limit2No health probes2No safeguard on destructive change2No staged rollout2Overlapping failures2Ineffective rollback2Unrepresentative staging2Credential in configuration1Cache-masked failure1No configuration tests1No resource requests or limits1No runbook1No headroom1Retry storm1Deferred upgrades1Gap in automated tests1
Organisation
Google Cloud3Microsoft Azure3Amazon Web Services2Binder Project2DFDS2Hyperconnect2Linode2Airbnb1AirMap1Argo CD users1Barush Mendez1Buffer1Cisco ThousandEyes1Cresta1Cycode1Devsisters1Grafana Labs1HubSpot1Inflab1Marcin Cuber1Monzo1Nebius1Nordstrom1Oath1Omio1Palark1Pinterest1PrometheusKube1Ravelin1Scaleway1Shopify1Skyscanner1thredUP1Toyota Connected1Wiz1Zalando1ZenML1
  1. IL-0058 Argo CD deleted namespaces shared by several applications, leaving about 97 applications stuck Missing and OutOfSync
    Argo CD users2026
  2. IL-0150 A latent race condition in DynamoDB's DNS automation left an empty DNS record for dynamodb.us-east-1.amazonaws.com, and the DynamoDB and EC2 launch failures that followed caused EKS, ECS and Fargate container launch failures and cluster scaling delays in us-east-1
    Amazon Web Services2025
  3. IL-0104 After a CoreDNS pod was OOMKilled, the talk names three causes: Istio's aggressive DNS cache refresh on one reused client port, kube-proxy not cleaning up conntrack entries of terminated pods, and the 2-minute UDP conntrack timeout
    Cisco ThousandEyes2025
  4. IL-0171 Karpenter node auto-repair replaced EKS GPU nodes about 10 minutes after they became Ready, in a loop, because the Bottlerocket NVIDIA AMI lacked libdcgm.so and set AcceleratedHardwareReady to False
    Inflab2025
  5. IL-0147 AKS nodes behind an HTTP proxy failed to provision with CSE exit code 31 on 202508 node images, because a refactoring stopped the proxy from being used when downloading uncached Kubernetes binaries
    Microsoft Azure2025
  6. IL-0247 One in a million search requests took 100 times longer on Pinterest's Manas leaf nodes after moving to Kubernetes
    Pinterest2025
  7. IL-0078 Enabling EKS secrets encryption with KMS through Terraform removed worker node IAM permissions and took down all ZenML Pro workspaces
    ZenML2025
  8. IL-0053 Unauthenticated ingress-nginx admission webhook let anyone on the pod network run code and read every Secret in the cluster (IngressNightmare, CVE-2025-1974)
    Wiz2025
  9. IL-0233 A VPC control plane release broke IP alias routes in Nebius eu-west1, and Managed Kubernetes pods lost connectivity
    Nebius2025
  10. IL-0081 API requests timed out during deployment rollouts because the ingress-nginx enable-serial-reloads option held back endpoint updates while a long NGINX reload was running
    Cresta2025
  11. IL-0157 After a release, new nodes could not join Linode Kubernetes Engine clusters with IP ACL enabled, so autoscaling failed on about 20 to 25 clusters
    Linode2025
  12. IL-0138 A failed Nvidia driver build left broken kernel packages on the only Kubernetes master node, which hit a kernel panic on reboot after a power outage and took the homelab cluster down for six days
    Barush Mendez2025
  13. IL-0347 About 25% of Scaleway FR-PAR-1 hypervisors lost VPC connectivity when both VPC BGP route reflectors got stuck
    Scaleway2024
  14. IL-0495 HubSpot's internal load balancers on Kubernetes were falsely marked unhealthy and the rest collapsed under the traffic
    HubSpot2024
  15. IL-0148 A kube-scheduler regression in Kubernetes v1.27.14, v1.28.10 and v1.29.5 made the scheduler panic on a pod whose node affinity named a nonexistent node, so no pods were scheduled on affected AKS clusters
    Microsoft Azure2024
  16. IL-0077 EKS cluster went read-only for over seven hours after Kyverno ephemeralreports filled the etcd database past its size limit
    Marcin Cuber2024
  17. IL-0161 A faulty earlier API update made new LKE cluster provisioning in Toronto time out for three days, leaving clusters with no node pools or the wrong number of nodes
    Linode2024
  18. IL-0173 cert-manager's preferredChain logic picked the Let's Encrypt chain through the expired DST Root CA X3, so clients built on an old BoringSSL could not connect to a development server
    Devsisters2024
  19. IL-0065 Any pod could take over the cluster by rewriting Argo CD's unauthenticated Redis cache (CVE-2024-31989)
    Cycode2024
  20. IL-0141 gke-metadata-server versions 0.4.272 to 0.4.280 flooded Cloud Logging with "Unable to sync sandbox" debug logs on GKE clusters with Workload Identity, using up logging quota and adding billable usage
    Google Cloud2023
  21. IL-0151 A latent defect in the AWS Lambda Frontend, set off when one us-east-1 cell scaled past a size never reached before, raised errors for Lambda and for services that depend on it, including new EKS cluster provisioning
    Amazon Web Services2023
  22. IL-0174 After an upgrade to Fluent Bit 2.0, logging sidecars that used the Kafka output plugin kept restarting with SIGSEGV because the plugin read a corrupted record as a map without checking its type
    Hyperconnect2023
  23. IL-0112 The daily node rollover put DFDS's EKS nodes on an AMI later recalled for a kernel issue, causing "PLEG is not healthy" NodeNotReady flaps and pod network timeouts
    DFDS2022
  24. IL-0139 Pods on GKE clusters with Calico network policy got stuck Terminating or Pending after upgrading to 1.22 or later because of a race condition in the Calico component
    Google Cloud2022
  25. IL-0175 Lowering a JVM pod's CPU request from 2 to 1 pushed memory past the 90% alert because the JVM treats a cgroup v1 cpu.shares value of exactly 1024 as unset and sized Reactor Netty threads from the node's 36 CPUs
    Hyperconnect2022
  26. IL-0403 mybinder.org could not create pods for hours as the GKE gke-resource-quotas quota wrongly reported 15k pods in use
    Binder Project2022
  27. IL-0462 Some MySQL pods on Shopify's GKE clusters took up to 30 minutes to start, slowing instance replacement
    Shopify2021
  28. IL-0106 A malformed Traefik IngressRoute triggered a Traefik v2 bug once a node rollover restarted every Traefik pod, and all Kubernetes services returned 404
    DFDS2021
  29. IL-0032 Mutating admission controller generated broken JSON patches, silently blocking deploys for a week until a production service ran down to two pods
    Airbnb2020
  30. IL-0146 New AKS clusters with Azure CNI in Central India lost tunnel connectivity because VMs began getting MAC addresses with a 06 prefix, which exposed a dormant bug in bridge-mode Azure CNI
    Microsoft Azure2020
  31. IL-0041 fluent-bit silently stopped shipping application logs from one node to Elasticsearch
    PrometheusKube2020
  32. IL-0029 Slow user-facing services from CPU throttling on containers that used far less CPU than their limit
    Buffer2020
  33. IL-0190 Cortex distributors kept a stuck connection to a dead etcd leader after a Kubernetes upgrade terminated its node, failing writes to Hosted Prometheus for 12 minutes
    Grafana Labs2020
  34. IL-0196 Adding a second Redis replica and lowering memory limits in one RedisFailover change restarted the only Redis pod that held data, and Sentinel promoted the empty new replica, erasing the data
    Palark2020
  35. IL-0012 CPU limits and an unpatched kernel CFS quota bug throttled multi-threaded services into stalls, failed health checks and timeouts
    Omio2020
  36. IL-0143 GKE nodes upgraded to certain 1.13 and 1.14 versions had an elevated number of kernel panics, fixed by releases that downgraded the Container-Optimized OS image
    Google Cloud2019
  37. IL-0062 Intermittent 502 errors behind a GKE Ingress because pods shut down before the load balancer stopped sending them traffic
    Ravelin2019
  38. IL-0011 HAProxy ingress piled requests onto the same pods because a Go template visited the backend map in sorted pod IP order
    Skyscanner2019
  39. IL-0030 Most nodes of a cluster went NotReady together for 15 minutes while the API server sat behind an AWS ELB
    Zalando2019
  40. IL-0050 Pods on Kubernetes failed to resolve an external Redis hostname because of ndots:5, the musl resolver in Alpine images, and Netty's DNS resolver
    Toyota Connected2018
  41. IL-0051 A third-party log shipping client leaked file descriptors with every new container until nodes failed with too many open files
    thredUP2018
  42. IL-0074 All AirMap platform services down as worker nodes went NotReady and pods crash-looped on network and DNS failures
    AirMap2018
  43. IL-0504 mybinder.org launches failed for about five hours as undeleted user pods piled up and filled the cluster
    Binder Project2018
  44. IL-0038 kubelet rollout hit a Kubernetes bug and left every node in a cluster NotReady, triggering mass pod eviction
    Oath2018
  45. IL-0072 High etcd write latency made the API servers return stale data, and the production cluster flipped between two views of its state
    Nordstrom2017
  46. IL-0003 linkerd lost Kubernetes service discovery after an etcd reconfiguration, and restarting every linkerd instance turned failed payments into a full platform outage
    Monzo2017

Faulty deployment20 cases

Component
Node pool7CI system4Ingress controller3Telemetry agent3Cloud load balancer2Cluster autoscaler2ConfigMaps and Secrets2Helm2Admission webhook1In-memory cache1Calico1Container CPU limit1Database1Descheduler1GPU driver1Model inference server1Managed Kubernetes control plane and provider API1Network control plane (SDN)1Node root disk1Node operating system image1Pod termination1Progressive delivery controller1Container registry and its credentials1Service discovery1
Contributing factor
Unverified assumption5Excessive blast radius4No alert on limit3Unreviewed defaults2Limited debug access2Monitoring blind spot2No staged rollout2Overlapping failures2Missed in review2Single point of failure2Unrepresentative staging2Gap in automated tests2Signal buried in noise1Automation amplification1Change started before off-hours1Change with a delayed effect1Unknown limit1Harmful manual action1No configuration tests1No resource requests or limits1No runbook1No headroom1Partial rollout1Ineffective rollback1Dependence on a warm cache1
Organisation
Linode5Google Cloud2Kubernetes project2UK Ministry of Justice2Airbnb1CircleCI1Gitpod1Gunosy1IONOS Cloud1Lucidworks1Microsoft Azure1Wikimedia Foundation1Yonghyun Kim1
  1. IL-0237 A Kubernetes patch upgrade evicted every Ray Serve head and worker pod at once, and Lucidworks hosted LLMs went down
    Lucidworks2026
  2. IL-0166 A compute host update removed the credential parser used to decode StackScripts, so new Linodes and every new LKE and LKE-E node failed to boot and autoscaling and node recycling failed in all regions
    Linode2026
  3. IL-0234 An onboarding operator deployed without its scope file installed dev applications on other IONOS Kubernetes clusters
    IONOS Cloud2026
  4. IL-0167 A phased rollout of an LKE software release in Osaka stalled LKE-E cluster creation, version upgrades and control plane ACL changes
    Linode2026
  5. IL-0160 A repaired legacy GPU monitoring script sent a firmware query that triggers a race condition in the GPU System Processor of NVIDIA RTX 4000 Ada GPUs, locking up GPU Linodes and LKE worker nodes in several regions
    Linode2026
  6. IL-0188 Rolling updates on EKS returned 502 errors for about 30 seconds because Kubernetes deleted old pods as soon as new pods passed readiness, before the ALB had marked the new pod targets healthy
    Yonghyun Kim2025
  7. IL-0020 During an EKS 1.30 upgrade the descheduler tried to reschedule every pod on the cordoned nodes at once and production workloads went down
    UK Ministry of Justice2025
  8. IL-0324 Three MediaWiki deployments lost Memcached after a PHP 8.1 rollout and overloaded the External Storage database cluster
    Wikimedia Foundation2025
  9. IL-0530 Virtual router controllers in us-central1 ran out of memory in a loop, cutting networking for new and migrated VMs
    Google Cloud2023
  10. IL-0142 GKE clusters on 1.24 and later wrote unexpected fluentbit error entries such as "Failed to get record: decoder: failed to decode payload: EOF" that used up the Logging API write quota for some customers
    Google Cloud2023
  11. IL-0437 Old pods from an unfinished Argo Rollouts canary could not start with a Helm-updated ConfigMap and died in a cascade
    Gunosy2023
  12. IL-0163 An API change that slipped through a gap in automated testing made LKE clusters resize to the wrong number of nodes and added billable worker nodes to customer accounts
    Linode2023
  13. IL-0164 An incremental Linode Kubernetes Engine production update that was missing changes from the previous release made cluster upgrades and node additions fail silently
    Linode2022
  14. IL-0591 Gitpod workspaces in all regions became IO bound and unresponsive for days after the ws39 workspace cluster release
    Gitpod2022
  15. IL-0086 A deployment meant for a CircleCI test Kubernetes cluster also removed Traefik and Calico resources from the production cluster, taking down the CircleCI UI and runner.circleci.com
    CircleCI2021
  16. IL-0088 A validating admission webhook added by a new ingress controller module version made the live-1 cluster reject every new Ingress with an x509 error
    UK Ministry of Justice2020
  17. IL-0034 A crashing stats agent sidecar marked every pod unready and the service mesh pulled all services out of traffic
    Airbnb2019
  18. IL-0149 Upgrades of AKS clusters on Virtual Machine Scale Sets failed with a 'platform image is not available' error after a service update left a backend service unhealthy
    Microsoft Azure2019
  19. IL-0467 Boskos crash-looped after a node drain started it from a buggy StatefulSet image, failing Kubernetes PR jobs for 9 hours
    Kubernetes project2019
  20. IL-0466 Prow scheduled test job pods onto its service cluster, leaving hundreds Pending and starving Prow's own services
    Kubernetes project2019

Certificate or credential expiry8 cases

Component
Kubernetes API server3Node pool3kubelet2Admission webhook1Calico1cert-manager1Cluster autoscaler1Cluster cloud identity1CNI plugin1etcd1GKE control plane1Managed Kubernetes control plane and provider API1Persistent volumes and stateful sets1Policy engine1RBAC and service accounts1Secrets store1
Contributing factor
Monitoring blind spot3Unreviewed defaults2Single point of failure2Unverified assumption1Change with a delayed effect1Ignored early signal1Unknown limit1Limited debug access1Harmful manual action1No alert on limit1Deferred upgrades1
Organisation
Linode2ckck8031dbhang1DFDS1GoCardless1Jenkins project1Webhook Relay1
  1. IL-0158 An expired credential caused authentication failures that stopped LKE and LKE-E from deploying clusters and nodes in all regions, which also blocked node recycling and cluster autoscaling
    Linode2026
  2. IL-0183 kubeadm's one-year leaf certificates expired, so kube-apiserver could not reach etcd, restarted in a loop and left the cluster unreachable
    ckck8032026
  3. IL-0156 Revoking old certificates after a rotation left a certificate key mismatch between API endpoints and the cluster controller, and LKE cluster management and autoscaling in Atlanta returned 5xx errors
    Linode2025
  4. IL-0107 Kyverno stuck in a crash loop behind a webhook with failurePolicy fail blocked every change to Kubernetes Service objects for 88 minutes
    DFDS2022
  5. IL-0113 The managed GKE control plane of Webhook Relay's main cluster failed because its API server certificates had never been rotated, and a manual credential rotation reported success without taking effect
    Webhook Relay2022
  6. IL-0521 AKS pods could not mount volumes and new nodes got no CNI network interface after the cluster service principal expired
    Jenkins project2021
  7. IL-0184 JupyterHub's KubeSpawner could not create containers and kubectl was refused because the API server and other internal certificates of a kubeadm cluster had expired
    dbhang2021
  8. IL-0460 Vault served an expired TLS certificate and almost every GoCardless service pod crash-looped, a 22-minute full outage
    GoCardless2020

Credential exposure1 case

Component
Kubernetes API serverRBAC and service accounts
Contributing factor
Excessive blast radius
Organisation
Unnamed cryptocurrency exchange
  1. IL-0069 Attackers stole a high-privilege service account token from a production Kubernetes cluster and used it to reach a cryptocurrency exchange's financial systems
    Unnamed cryptocurrency exchange2026

Breaking change in an upgrade19 cases

Component
Ingress controller6Node operating system image6Admission webhook5Cloud load balancer3Client-side DNS resolver and cache3cert-manager2ConfigMaps and Secrets2GKE control plane2Helm2Istio2kube-proxy2kubelet2Linux kernel2Node pool2Calico1CDN and edge network1Cilium1Cluster autoscaler1CoreDNS1etcd1Model inference server1Karpenter1Kubernetes API server1Persistent volumes and stateful sets1Container registry and its credentials1systemd-networkd1Web application firewall rules1
Contributing factor
Change with a delayed effect5Monitoring blind spot5Unrepresentative staging4No staged rollout3Ineffective rollback3Deferred upgrades3Unverified assumption2Automation amplification2Harmful manual action2Single point of failure2Untested backup1Excessive blast radius1Configuration not in code1Unreviewed defaults1Ignored early signal1Cache-masked failure1Unpatched known bug1Limited debug access1No headroom1Partial rollout1Unpinned version1
Organisation
2i2c3PortSwigger2UK Ministry of Justice2Altinity1CircleCI1Datadog1Enigmo1GitHub1Honeycomb1Jenkins project1Microsoft Azure1Reddit1Sunwoo Park1Wealthy1Wikimedia Foundation1
  1. IL-0097 A cluster autoscaler version bump was incompatible with the one cluster still on Kubernetes 1.33, so no scale-up events fired and new nodes were not created
    2i2c2026
  2. IL-0098 After 2i2c moved from ingress-nginx to nginx-ingress, a hub's Ingress took priority over cert-manager's temporary Ingress for the same host, so new hubs could not get TLS certificates
    2i2c2026
  3. IL-0092 Upgrading ingress-nginx from 1.12.0 to 1.14.3 brought a newer ModSecurity and OWASP Core Rule Set whose false positives blocked HMPPS Auth, so users could not sign in to their applications
    UK Ministry of Justice2026
  4. IL-0236 After an Istio 1.15 to 1.24 upgrade, half of Wikimedia's ML model servers returned 5XX for about a day
    Wikimedia Foundation2026
  5. IL-0405 An expired AWS Load Balancer Controller webhook certificate left no healthy targets and took BSEE SaaS down
    PortSwigger2026
  6. IL-0101 After Google forced a GKE control plane upgrade to 1.33.2 at a deprecation deadline, no new node could initialise, so autoscaling and spot node replacement stopped for 15 hours
    Wealthy2025
  7. IL-0079 A GKE node auto-upgrade to an ubuntu_containerd image with a kernel regression left kube-proxy unable to apply its ip6tables rules, so load balancer health checks failed and several clusters lost all ingress traffic
    Altinity2025
  8. IL-0201 A framework upgrade caused kernel panics in GitHub's Kubernetes infrastructure, and GitHub.com web and API requests, mostly for issues and pull requests, failed intermittently for about 16 hours
    GitHub2025
  9. IL-0404 All BSEE SaaS customer pods on EKS crash-looped after Karpenter moved nodes to Bottlerocket v1.26.0, which broke Java
    PortSwigger2024
  10. IL-0203 A routine Kubernetes cluster system update made the AWS Load Balancer Controller fail, so new pods were never registered with the ALB and Honeycomb's EU1 ingest and UI went down once the last old pods were replaced
    Honeycomb2024
  11. IL-0180 A GKE control plane auto-upgrade from 1.22 to 1.23 removed the v1beta1 Ingress and ValidatingWebhookConfiguration APIs that Istio 1.7.4 still used, and Istio stopped working in production
    Sunwoo Park2023
  12. IL-0060 Kubernetes 1.24 upgrade removed the master node label that Calico route reflectors selected on, dropping all pod routes and taking Reddit down for 314 minutes
    Reddit2023
  13. IL-0230 kube-proxy sync failures during a Kubernetes upgrade corrupted node iptables, and CircleCI jobs stopped starting
    CircleCI2023
  14. IL-0198 An automatic Ubuntu security update restarted systemd-networkd, which flushed the IP rules Cilium relied on and cut more than 60 percent of Datadog's Kubernetes nodes off the network across regions and cloud providers
    Datadog2023
  15. IL-0144 A systemd security update installed by unattended-upgrades lost the DNS configuration of Ubuntu 18.04 VMs, so AKS kubelets could not resolve registry names and pods failed with ImagePullBackOff
    Microsoft Azure2022
  16. IL-0501 Every node of the UToronto Kubernetes cluster on Azure went NotReady and the JupyterHub became unreachable
    2i2c2022
  17. IL-0522 An ingress-nginx chart upgrade deleted the jenkins.io Ingress and Fastly served HTTP/503 errors for hours
    Jenkins project2021
  18. IL-0090 After an upgrade of the ModSec ingress controller module, its validating webhook served a certificate for the wrong service name and blocked every Ingress create and change
    UK Ministry of Justice2021
  19. IL-0449 Pods on Enigmo's new EKS managed node groups kept restarting or stuck Terminating and the nodes became unusable
    Enigmo2020

Data loss or corruption1 case

Component
Container registry and its credentials
Contributing factor
Cache-masked failure
Organisation
Harness
  1. IL-0474 Harness Feature Flags auth returned 502s when rescheduled pods could not get an image deleted from Artifact Registry
    Harness2025

Operator error9 cases

Component
Ingress controller4Persistent volumes and stateful sets3CI system2Database2Node pool2Terraform2Cloud load balancer1Credential broker for pods1Elasticsearch or OpenSearch1external-dns1Helm1Jobs and CronJobs1JupyterHub1kube-proxy1Node launch template1Model hub1Prometheus1Container registry and its credentials1VPC route tables and firewall rules1
Contributing factor
No safeguard on destructive change6Unverified assumption2Credential in configuration1Unreviewed defaults1Harmful manual action1No runbook1No headroom1Overlapping failures1Shared credential1Deferred upgrades1Unsafe failure behaviour1
Organisation
UK Ministry of Justice32i2c1DevOps Hof1GitLab1OpenAI1Palark1UC Berkeley1
  1. IL-0095 An engineer deleted the prod namespace of the LEAP JupyterHub while believing the kubeconfig pointed at a new cluster, and the hub was down for about an hour
    2i2c2025
  2. IL-0244 A queued pull request destroyed the EKS subnet route table associations and Cloud Platform services went offline
    UK Ministry of Justice2024
  3. IL-0018 A Docker Hub password rotation left EKS nodes and Concourse locked out of Docker Hub and unable to pull images
    UK Ministry of Justice2023
  4. IL-0207 A routine task run from an engineer's local environment switched to production and deleted GitLab.com's Consul release, so Patroni lost Consul and GitLab.com returned 5xx errors to all traffic
    GitLab2022
  5. IL-0115 One of OpenAI's three production AKS clusters went offline through a human error, and rebuilding it took 10½ hours because an AKS Kubernetes upgrade broke the ingress and AKS cannot be downgraded
    OpenAI2022
  6. IL-0197 Moving a three-node Elasticsearch StatefulSet to new persistent volumes split the cluster in two, because the Helm chart hardcoded cluster.initial_master_nodes to elasticsearch-0
    Palark2021
  7. IL-0089 A test cluster deletion script run with the kube context pointing at the live-1 production cluster deleted its ingress controller, monitoring, logging, cert-manager, kiam and external-dns
    UK Ministry of Justice2020
  8. IL-0054 All ingress traffic lost while draining a GKE node pool because the load balancer only targeted new nodes without ingress-nginx pods
    DevOps Hof2019
  9. IL-0506 UC Berkeley DataHub logins failed and students got empty home directories after the JupyterHub database was overwritten
    UC Berkeley2017

unknown5 cases

Component
Container runtime2Calico1etcd1Kubernetes API server1kubelet1Multus meta-CNI1Node pool1Persistent volumes and stateful sets1Prometheus1
Contributing factor
Harmful manual action2Unverified assumption1Unreviewed defaults1Ignored early signal1Monitoring blind spot1Pages not delivered1Single point of failure1
Organisation
Binder Project1Fyra Labs1NRE Labs1UC Berkeley1UK Ministry of Justice1
  1. IL-0045 Prometheus on the live cluster restarted several times after readiness probe failures, leaving gaps in metrics
    UK Ministry of Justice2023
  2. IL-0087 After a cold reboot of RKE2 nodes, pods could not get IP addresses because Multus could not reach the Kubernetes API, and further reboots spread the failure to every node
    Fyra Labs2023
  3. IL-0059 Single-master Kubernetes control plane kept dying because the kubelet restarted a healthy etcd on a failing kubeadm liveness probe
    NRE Labs2018
  4. IL-0402 mybinder.org launches dropped to zero for hours because a JupyterHub pod on a lost GKE node kept its hub database PVC
    Binder Project2018
  5. IL-0507 Student pods on UC Berkeley DataHub got stuck in Terminating or would not start because Docker hung on two Azure nodes
    UC Berkeley2017

Hardware failure1 case

Component
ConfigMaps and SecretsetcdKubesprayNode root disk
Contributing factor
Single point of failure
Organisation
Hans H. Kim
  1. IL-0185 A disk failure made the first control plane node of a Kubespray cluster read-only, and recovery needed the Kubespray inventory reordered and a new control plane node joined
    Hans H. Kim2023