Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases71 cases

31 to 35 of 71

  1. IL-0091 Changing the worker node instance type through Terraform recycled all nodes at once and deleted users' production pods together
    UK Ministry of Justice2023
    Symptoms

    At 10:23 on 6 June 2023 a user of the Cloud Platform reported that all of their production pods had been deleted at the same moment. Several users received Pingdom alerts that their applications were down for a few minutes.

    Service outage
    Trigger

    The team changed the instance type of the cluster's worker nodes through Terraform.

    Root cause
    Configuration change
    Component
    Node poolTerraform
    Contributing factor
    Excessive blast radiusNo runbook
    Detection

    Users reported that their production pods had been deleted at once and that Pingdom alerts showed their applications down; the incident was declared at 11:00.

    Action items
    • Ask AWS, through a support ticket, for the steps to change the node instance type without an outage
    • Plan how to make instance type changes through Terraform without downtime and add a runbook with the steps for changing the node instance type
    View case
  2. IL-0180 A GKE control plane auto-upgrade from 1.22 to 1.23 removed the v1beta1 Ingress and ValidatingWebhookConfiguration APIs that Istio 1.7.4 still used, and Istio stopped working in production
    Sunwoo Park2023
    Symptoms

    On 2023-05-18 a Slack alert arrived at 20:01 and Istio stopped working properly in the production GKE cluster. The istiod error logs said that the v1beta1.Ingress and v1beta1.ValidatingWebhookConfiguration resources could not be found.

    Service outage
    Trigger

    GKE automatically upgraded the production cluster's control plane from 1.22.17-gke.7500 to 1.23.17-gke.1700 at 19:52 on 2023-05-18.

    Root cause
    Breaking change in an upgrade
    Component
    IstioGKE control planeAdmission webhook
    Contributing factor
    Deferred upgrades
    Detection

    A Slack alert at 20:01 on 2023-05-18, nine minutes after the control plane upgrade. The istiod error logs showed that the v1beta1 Ingress and ValidatingWebhookConfiguration resources could not be found.

    Remediation
    • upgraded Istio from 1.7.4 to 1.13.3
    Lessons

    The first diagnosis, end of support for Istio, was wrong; the real cause was found by matching the alert time against the cluster's UPGRADE_MASTER operations in Logs Explorer. The author recommends checking the version compatibility matrices of Kubernetes, Istio and other packages, keeping a production version for at least six months and upgrading three months before end of support, keeping the test environment on the latest version, backing up before upgrades, managing deployments and versions with a tool such as Argo CD, and testing upgrades in a test environment before production.

    View case
  3. IL-0087 After a cold reboot of RKE2 nodes, pods could not get IP addresses because Multus could not reach the Kubernetes API, and further reboots spread the failure to every node
    Fyra Labs2023
    Symptoms

    At around 2 AM ICT on 2023-04-09 the team saw unusual CPU use on one cluster node, which throttled every other service on it and made them fail. They presumed an upstream container runtime bug in containerd and runc and decided to fully reboot the affected nodes, taking the cluster down for maintenance on the assumption that it was not in active use.

    Sustained high node CPUService outage
    Trigger

    A full cold reboot of the nodes that showed unusual CPU use, followed by upgrades and cold reboots of the other nodes.

    Root cause
    unknown
    Component
    Multus meta-CNICalicoContainer runtime
    Contributing factor
    Harmful manual actionUnverified assumption
    Detection

    The team saw unusual CPU use on a node around 2 AM ICT and failing services on the rebooted nodes around 2:30 ICT; developers reported the Terra repositories down at 6:31 ICT.

    Remediation
    • uninstalled Multus and Calico and replaced them with the Cilium CNI
    • rolled back most cluster settings except the CNI
    Lessons

    Rebooting nodes to clear one problem can expose another; here a CNI chain could not reach the API server after a cold start. Repeating the same reboot on more nodes while the first failure was not understood spread the outage from a few nodes to the whole cluster. The team also found that Cilium's real resource use in their cluster was acceptable, contrary to the benchmark they had relied on.

    View case
  4. IL-0060 Kubernetes 1.24 upgrade removed the master node label that Calico route reflectors selected on, dropping all pod routes and taking Reddit down for 314 minutes
    Reddit2023
    Symptoms

    About two minutes after an engineer started the Kubernetes 1.23 to 1.24 upgrade of the cluster running Reddit's legacy stack, just after 19:00 UTC, the site stopped. The CDN edge request graph, which is kept separate from the cluster, showed traffic by status collapsing. Users saw the error page, a homepage with no posts, or plain errors.

    Service outageHigh error rateDNS resolution failuresMonitoring outagePods stuck in ContainerCreatingControl plane lag
    Trigger

    An in-place upgrade of the legacy cluster from Kubernetes 1.23 to 1.24; in 1.24 the node-role.kubernetes.io/master label is removed even from running clusters.

    Root cause
    Breaking change in an upgrade
    Component
    CalicoKubernetes API serverAdmission webhooketcd
    Contributing factor
    Configuration not in codeUnrepresentative stagingUntested backupMonitoring blind spotSingle point of failure
    Detection

    The site broke about two minutes into the upgrade; an incident was opened immediately and responders were on the call within three minutes, working from CDN edge graphs because the cluster's own metrics were gone.

    Remediation
    • deleted the Open Policy Agent webhook configurations to stop API server timeouts
    • terminated all worker nodes of the cluster
    • restored the control plane from the pre-upgrade backup onto the first control plane node
    • switched off the cluster autoscaler until the control plane was stable
    • generated new TLS certificates so replacement control plane nodes could join etcd
    • brought traffic back in steps from 1% to 100%
    • restarted legacy services that had been stopped by hand
    Action items
    • Codify cluster configuration so every choice has a record, and document in detail what cannot be codified
    • Standardise components across clusters and replace bespoke configurations
    • Build tooling to manage the lifecycle of whole clusters so they can be recreated or replicated
    • Add pre-production clusters that can be tested with confidence
    • Improve the backup restore process and make cluster upgrades safe
    Lessons

    Configuration that exists only in a live cluster, edited by hand through a special tool, can be forgotten, as the route reflector setup was, and then break on an upgrade nobody connects it to. A Kubernetes upgrade cannot be reverted without a restore, so the restore procedure has to be kept current and practised. Consistent, codified clusters are what make upgrades testable; bespoke clusters keep producing outages.

    View case
  5. IL-0198 An automatic Ubuntu security update restarted systemd-networkd, which flushed the IP rules Cilium relied on and cut more than 60 percent of Datadog's Kubernetes nodes off the network across regions and cloud providers
    Datadog2023
    Symptoms

    Between 06:00 and 07:00 UTC on 8 March 2023, Kubernetes nodes in several Datadog regions lost network connectivity at the same time, on AWS, Google Cloud and Azure. All services across multiple regions were affected, including the regions that host Datadog's CI/CD and automation tools. Datadog's regions are not connected to each other and changes are rolled out one region at a time, so a simultaneous failure on several cloud providers looked implausible at first.

    Node NotReadyService outageThroughput degradation
    Trigger

    On 7 March 2023 Ubuntu published a systemd patch for a CVE (249.11-0ubuntu3.7). Ubuntu's default unattended upgrades installed it on the nodes between 06:00 and 07:00 UTC on 8 March, and installing it restarted systemd-networkd.

    Root cause
    Breaking change in an upgrade
    Component
    systemd-networkdCiliumNode operating system image
    Contributing factor
    Unreviewed defaultsChange with a delayed effectNo staged rolloutAutomation amplification
    Remediation
    • restarted thousands of disconnected instances on Google Cloud and Azure through the cloud APIs, keeping their local disks
    • rebuilt the data lost with terminated AWS instances' local disks through replication, reprocessing and restores from backup
    Lessons

    A change in a base operating system component can stay dormant for months until something restarts that component. Here the trigger was a security patch, and the change reached every region at once because automatic OS updates ran outside the progressive, region-by-region process used for every other change. Cloud providers react differently to an instance that loses its network. Google Cloud and Azure left the instances running, so a restart recovered them with their disks; AWS replaced them quickly, which brought stateless services back fast but destroyed the local-disk data the stateful services needed.

    View case
Check your cluster for this

These 71 cases involve Node pool and Ingress controller. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause11

Capacity exhaustion13 cases

Component
Node pool4CoreDNS3Application database3Container memory limit3In-memory cache2kubelet2Persistent volumes and stateful sets2Pod scheduling and placement2Admission webhook1Cloud controller manager1conntrack table1Container runtime1Client-side DNS resolver and cache1Helm1Horizontal pod autoscaler1Ingress controller1Istio1Jobs and CronJobs1kube-dns1Linux kernel1NAT gateway1Node root disk1Container registry and its credentials1
Contributing factor
Single point of failure7Automation amplification5Monitoring blind spot4Unverified assumption3Excessive blast radius3Unreviewed defaults3Retry storm3Unknown limit2No caching2Change during peak hours1Ignored early signal1Limited debug access1Harmful manual action1No capacity planning1No resource requests or limits1No runbook1No headroom1
Organisation
Backpackr22i2c1BC Government1Buildkite1Datadog1Honeycomb1Magic Pages1Mercari1Moonlight1RebelMouse1Saltside1Universe1
  1. IL-0102 CoreDNS pods on a production EKS cluster ran out of memory under a burst of new pods, endpoints and nodes, taking down the Buildkite web UI, APIs and job dispatch
    Buildkite2026
  2. IL-0067 Redis filled its persistent volume, corrupted its append-only file and crash-looped on every restart
    BC Government2025
  3. IL-0080 kube-dns on Mercari's GKE clusters could not keep up with peak-hour DNS query rates, and a Sev1 production outage followed when several services failed DNS resolution
    Mercari2025
  4. IL-0082 Losing one CoreDNS pod during a planned EKS node reroll overloaded the remaining CoreDNS pods, causing DNS timeouts and errors across multiple services at RebelMouse
    RebelMouse2025
  5. IL-0170 During a Lunar New Year event, traffic more than three times normal filled the nf_conntrack table on the EKS node running the Istio ingress gateway, dropping packets and taking down Backpackr's web and app services
    Backpackr2025
  6. IL-0027 All CoreDNS pods landed on one node, the node ran out of memory, and DNS for the whole service went down
    Backpackr2025
  7. IL-0085 A control plane node of Magic Pages' Kubernetes cluster on Hetzner Cloud ran out of memory, the cluster's own calls then used up the Hetzner API rate limit, and all customer websites were down for about 7.5 hours
    Magic Pages2025
  8. IL-0202 A burst of queries and database writes exhausted connection pools and memory on Honeycomb's ingest hosts, and Kubernetes crash-loop back-off kept failed hosts offline so ingest capacity stayed too low to recover
    Honeycomb2023
  9. IL-0094 Student pods filled the shared nodes of a JupyterHub cluster and starved the hub's core pods, freezing the hub
    2i2c2020
  10. IL-0037 Image registry rate-limited a whole region after a crash-looping DaemonSet with imagePullPolicy Always pulled through three NAT instances
    Datadog2019
  11. IL-0064 CPU-heavy pods packed onto one Google Cloud node drove it to kernel panics, and the crash moved from node to node until the site was down
    Moonlight2019
  12. IL-0036 A reindex Job that kept failing exhausted one node after another until the whole cluster was down
    Universe2017
  13. IL-0040 Parallel Helm installs saturated small gp2 root volumes on a kops cluster and left all three workers NotReady with the kubelet or Docker down
    Saltside2017

Dependency failure3 cases

Component
Data center power and cooling2Cloud block storage1etcd1Liveness and readiness probes1InfiniBand fabric1
Contributing factor
Overlapping failures2Single point of failure2Automation amplification1Excessive blast radius1Unreviewed defaults1Monitoring blind spot1No alert on limit1No runbook1
Organisation
Crusoe Cloud1Nebius1Yonghyun Kim1
  1. IL-0133 A storm shut down cooling at Nebius's us-central1 data center, and a disk hot-plug bug kept Managed Kubernetes control planes down for hours after power returned
    Nebius2026
  2. IL-0187 A brief Spring Cloud Config Server downtime failed every client application's liveness probe, because the actuator health check included the config server, and Kubernetes restarted the pods in a kill loop
    Yonghyun Kim2025
  3. IL-0135 A power source switch during UPS maintenance cut power to a row of GPU servers and shut down Crusoe Cloud VMs in us-northcentral1
    Crusoe Cloud2023

Breaking change in an upgrade7 cases

Component
Admission webhook2Cloud load balancer2GKE control plane2Ingress controller2Node operating system image2Calico1Cilium1ConfigMaps and Secrets1Client-side DNS resolver and cache1etcd1Istio1Kubernetes API server1kube-proxy1Linux kernel1Node pool1Persistent volumes and stateful sets1systemd-networkd1Web application firewall rules1
Contributing factor
Unrepresentative staging3Change with a delayed effect2Monitoring blind spot2No staged rollout2Single point of failure2Deferred upgrades2Automation amplification1Untested backup1Configuration not in code1Unreviewed defaults1Unpatched known bug1No headroom1Ineffective rollback1
Organisation
Altinity1Datadog1Honeycomb1Reddit1Sunwoo Park1UK Ministry of Justice1Wealthy1
  1. IL-0092 Upgrading ingress-nginx from 1.12.0 to 1.14.3 brought a newer ModSecurity and OWASP Core Rule Set whose false positives blocked HMPPS Auth, so users could not sign in to their applications
    UK Ministry of Justice2026
  2. IL-0101 After Google forced a GKE control plane upgrade to 1.33.2 at a deprecation deadline, no new node could initialise, so autoscaling and spot node replacement stopped for 15 hours
    Wealthy2025
  3. IL-0079 A GKE node auto-upgrade to an ubuntu_containerd image with a kernel regression left kube-proxy unable to apply its ip6tables rules, so load balancer health checks failed and several clusters lost all ingress traffic
    Altinity2025
  4. IL-0203 A routine Kubernetes cluster system update made the AWS Load Balancer Controller fail, so new pods were never registered with the ALB and Honeycomb's EU1 ingest and UI went down once the last old pods were replaced
    Honeycomb2024
  5. IL-0180 A GKE control plane auto-upgrade from 1.22 to 1.23 removed the v1beta1 Ingress and ValidatingWebhookConfiguration APIs that Istio 1.7.4 still used, and Istio stopped working in production
    Sunwoo Park2023
  6. IL-0060 Kubernetes 1.24 upgrade removed the master node label that Calico route reflectors selected on, dropping all pod routes and taking Reddit down for 314 minutes
    Reddit2023
  7. IL-0198 An automatic Ubuntu security update restarted systemd-networkd, which flushed the IP rules Cilium relied on and cut more than 60 percent of Datadog's Kubernetes nodes off the network across regions and cloud providers
    Datadog2023

Configuration change17 cases

Component
Configuration templating4Node pool4Cloud load balancer3Terraform3Argo CD2Kubernetes API server2kubelet2Container memory limit2Persistent volumes and stateful sets2Telemetry agent2AWS VPC CNI2Admission webhook1In-memory cache1CNI plugin1ConfigMaps and Secrets1Container runtime1Client-side DNS resolver and cache1GKE control plane1Ingress controller1Jobs and CronJobs1Node launch template1Linux kernel1Node operating system image1Pod priority and preemption1Service discovery1
Contributing factor
Automation amplification6Monitoring blind spot5Unverified assumption4Excessive blast radius4Unreviewed defaults4Missed in review4Unrepresentative staging4No configuration tests3No runbook3No staged rollout3Change with a delayed effect2Limited debug access2No health probes2No safeguard on destructive change2No headroom2Retry storm2Untested backup1Ignored early signal1Cache-masked failure1Harmful manual action1Overlapping failures1Partial rollout1Single point of failure1Unpinned version1
Organisation
DFDS3Grafana Labs3OpenAI2AdGuard VPN1CoW Protocol1DigitalOcean1InfluxData1Jetstack1Pusher1Skyscanner1Target1UK Ministry of Justice1
  1. IL-0063 VPN outage after recreating a Redis StatefulSet restarted the whole Redis cluster without readiness checks
    AdGuard VPN2025
  2. IL-0192 A Crossplane composition merge-policy change for a TLS policy update switched Kubernetes Services from LoadBalancer to ClusterIP and deleted the load balancers behind about 25% of Grafana Cloud services
    Grafana Labs2025
  3. IL-0005 A new telemetry service overloaded the Kubernetes API servers of large clusters and broke DNS-based service discovery for all OpenAI services
    OpenAI2024
  4. IL-0114 An unpinned AWS VPC CNI plugin was updated automatically during a capacity expansion, leaving new pods Pending without IP addresses and stopping CoW Protocol settlements for about 3.5 hours
    CoW Protocol2024
  5. IL-0153 DOKS worker nodes went NotReady at 06:00 UTC on three days because a systemd timer in the Debian image ran apt upgrade and killed it mid kernel upgrade, breaking node networking
    DigitalOcean2023
  6. IL-0091 Changing the worker node instance type through Terraform recycled all nodes at once and deleted users' production pods together
    UK Ministry of Justice2023
  7. IL-0109 A retried node rollover on a DFDS EKS cluster set the Auto Scaling group back to 15 instances, and the default termination policy removed every healthy worker node, leaving the cluster with no nodes
    DFDS2023
  8. IL-0116 A CNI configuration change incompatible with a hardware variant that exists only in production cut network connectivity in an OpenAI GPU cluster and took text-davinci-003 fully down
    OpenAI2023
  9. IL-0191 A label selector accidentally added to the Loki distributor Service matched only the new pods of a rollout, which took all write traffic, were OOM-killed and stalled the rollout
    Grafana Labs2022
  10. IL-0009 A duplicated generated name made Argo CD replace the InfluxDB Cloud production Application and delete the whole production environment
    InfluxData2022
  11. IL-0111 Switching the main Kubernetes ingress controller from Traefik v1 to v2 stopped X-Forwarded headers being passed, which broke the Okta login flow for every service using Okta and stopped DFDS online bookings for over an hour
    DFDS2021
  12. IL-0010 A templating change that dropped its {{ }} delimiters made Argo CD delete all 478 Skyscanner services in every region
    Skyscanner2021
  13. IL-0105 Replacing the AWS VPC CNI during a Terraform refactoring detached the ALB target groups, and every service in the Kubernetes cluster returned 503
    DFDS2021
  14. IL-0008 An unresponsive OPA validating webhook crash-looped the API server during a GKE master upgrade, and node auto-repair then recreated every node
    Jetstack2019
  15. IL-0004 A new default pod priority let a new Cortex cluster preempt every production ingester in a cascade, taking Grafana Cloud Hosted Prometheus down
    Grafana Labs2019
  16. IL-0066 Broken ConfigMap change lay dormant for weeks, then took down Pusher's edge proxy when a cluster upgrade replaced its pods
    Pusher2019
  17. IL-0055 A failing CronJob with the default concurrency policy piled up thousands of restarting pods and took down a cluster
    Target2018

Faulty deployment4 cases

Component
Ingress controller2Calico1Cloud load balancer1Cluster autoscaler1Descheduler1Node pool1Pod termination1Service discovery1Telemetry agent1
Contributing factor
Unverified assumption2Excessive blast radius2Unreviewed defaults2Monitoring blind spot2No alert on limit1No runbook1No headroom1
Organisation
Airbnb1CircleCI1UK Ministry of Justice1Yonghyun Kim1
  1. IL-0188 Rolling updates on EKS returned 502 errors for about 30 seconds because Kubernetes deleted old pods as soon as new pods passed readiness, before the ALB had marked the new pod targets healthy
    Yonghyun Kim2025
  2. IL-0020 During an EKS 1.30 upgrade the descheduler tried to reschedule every pod on the cordoned nodes at once and production workloads went down
    UK Ministry of Justice2025
  3. IL-0086 A deployment meant for a CircleCI test Kubernetes cluster also removed Traefik and Calico resources from the production cluster, taking down the CircleCI UI and runner.circleci.com
    CircleCI2021
  4. IL-0034 A crashing stats agent sidecar marked every pod unready and the service mesh pulled all services out of traffic
    Airbnb2019

Operator error5 cases

Component
Ingress controller4Cloud load balancer1Credential broker for pods1Application database1external-dns1Jobs and CronJobs1kube-proxy1Model hub1Node pool1Persistent volumes and stateful sets1Prometheus1Terraform1
Contributing factor
No safeguard on destructive change4Unverified assumption1Unreviewed defaults1No runbook1No headroom1Overlapping failures1Deferred upgrades1
Organisation
2i2c1DevOps Hof1GitLab1OpenAI1UK Ministry of Justice1
  1. IL-0095 An engineer deleted the prod namespace of the LEAP JupyterHub while believing the kubeconfig pointed at a new cluster, and the hub was down for about an hour
    2i2c2025
  2. IL-0207 A routine task run from an engineer's local environment switched to production and deleted GitLab.com's Consul release, so Patroni lost Consul and GitLab.com returned 5xx errors to all traffic
    GitLab2022
  3. IL-0115 One of OpenAI's three production AKS clusters went offline through a human error, and rebuilding it took 10½ hours because an AKS Kubernetes upgrade broke the ingress and AKS cannot be downgraded
    OpenAI2022
  4. IL-0089 A test cluster deletion script run with the kube context pointing at the live-1 production cluster deleted its ingress controller, monitoring, logging, cert-manager, kiam and external-dns
    UK Ministry of Justice2020
  5. IL-0054 All ingress traffic lost while draining a GKE node pool because the load balancer only targeted new nodes without ingress-nginx pods
    DevOps Hof2019

Software defect12 cases

Component
Kubernetes API server3Cluster autoscaler2Application database2etcd2Ingress controller2kubelet2Container memory limit2Model request routing2Persistent volumes and stateful sets2Cloud load balancer1GPU driver1Linkerd1Linux kernel1Model hub1Node pool1Telemetry agent1Terraform1
Contributing factor
Single point of failure3Unverified assumption2Excessive blast radius2Change with a delayed effect2Monitoring blind spot2No headroom2Overlapping failures2Automation amplification1Unreviewed defaults1Ignored early signal1Unpatched known bug1Harmful manual action1No alert on limit1No configuration tests1No staged rollout1Retry storm1Ineffective rollback1Deferred upgrades1Unrepresentative staging1
Organisation
AirMap1Barush Mendez1DFDS1Google Cloud1Hugging Face1Linode1Monzo1Nordstrom1OpenAI1Palark1thredUP1ZenML1
  1. IL-0078 Enabling EKS secrets encryption with KMS through Terraform removed worker node IAM permissions and took down all ZenML Pro workspaces
    ZenML2025
  2. IL-0157 After a release, new nodes could not join Linode Kubernetes Engine clusters with IP ACL enabled, so autoscaling failed on about 20 to 25 clusters
    Linode2025
  3. IL-0138 A failed Nvidia driver build left broken kernel packages on the only Kubernetes master node, which hit a kernel panic on reboot after a power outage and took the homelab cluster down for six days
    Barush Mendez2025
  4. IL-0024 Expensive cache-backed API calls and uncancelled database requests overloaded MongoDB and took the Hugging Face Hub down three times
    Hugging Face2024
  5. IL-0123 A response buffer allocated anew in a loop made OpenAI's routing layer nodes hit their memory limits and fail readiness checks on a morning with more completions than any day before, causing 502 and 503 errors across all models
    OpenAI2023
  6. IL-0132 A faulty resource cleanup process marked Vertex AI prediction endpoints as deleted worldwide, undeploying customers' models and failing predictions for 4 hours 20 minutes
    Google Cloud2022
  7. IL-0106 A malformed Traefik IngressRoute triggered a Traefik v2 bug once a node rollover restarted every Traefik pod, and all Kubernetes services returned 404
    DFDS2021
  8. IL-0196 Adding a second Redis replica and lowering memory limits in one RedisFailover change restarted the only Redis pod that held data, and Sentinel promoted the empty new replica, erasing the data
    Palark2020
  9. IL-0051 A third-party log shipping client leaked file descriptors with every new container until nodes failed with too many open files
    thredUP2018
  10. IL-0074 All AirMap platform services down as worker nodes went NotReady and pods crash-looped on network and DNS failures
    AirMap2018
  11. IL-0072 High etcd write latency made the API servers return stale data, and the production cluster flipped between two views of its state
    Nordstrom2017
  12. IL-0003 linkerd lost Kubernetes service discovery after an etcd reconfiguration, and restarting every linkerd instance turned failed payments into a full platform outage
    Monzo2017

Cloud provider outage6 cases

Component
Application database3Cluster autoscaler2Node pool2AKS control plane1GKE control plane1Helm1Horizontal pod autoscaler1Ingress controller1Kubernetes API server1Node root disk1Persistent volumes and stateful sets1
Contributing factor
Overlapping failures3Unverified assumption2Excessive blast radius2Single point of failure2Ignored early signal1Unpatched known bug1No capacity planning1No headroom1Pages not delivered1Unrepresentative staging1
Organisation
2i2c1Devsisters1Grafana Labs1loveholidays1Microsoft Azure1Namespace1
  1. IL-0084 Packet drops on a bare metal provider's vSwitch forced Namespace to rebuild its fra3 region and left global capacity short, and the next day slow NVMe writes on a control plane node drove the Kubernetes API server to very high CPU
    Namespace2024
  2. IL-0145 A power sag and chiller failure in one Australia East availability zone took down the AKS control plane's compute and its non-zone-redundant SQL Database operation queue, blocking AKS create, update and delete operations in Australia East and Australia Southeast
    Microsoft Azure2023
  3. IL-0096 A GKE node pool pinned to the europe-west2-b zone could not scale for 6.5 hours because Google ran out of resources in that zone (ZONE_RESOURCE_POOL_EXHAUSTED)
    2i2c2022
  4. IL-0172 An AWS Tokyo data centre cooling failure took down 6 of 60 CockroachDB nodes on Kubernetes within 22 minutes, and ranges lost quorum because the Helm-deployed cluster had no availability zone locality configured
    Devsisters2021
  5. IL-0193 A GCP persistent disk degradation took down a Cassandra StatefulSet whose OrderedReady policy blocked recovery, and Metrictank could not reconnect to the recreated pods, causing a 23-hour Graphite query outage
    Grafana Labs2019
  6. IL-0014 A GKE upgrade to a withdrawn version stalled on slow drains and a stuck node pool, then broke an ingress and confused the HPA
    loveholidays2019

unknown2 cases

Component
Calico1Container runtime1etcd1Kubernetes API server1kubelet1Multus meta-CNI1
Contributing factor
Harmful manual action2Unverified assumption1Unreviewed defaults1Ignored early signal1Monitoring blind spot1Single point of failure1
Organisation
Fyra Labs1NRE Labs1
  1. IL-0087 After a cold reboot of RKE2 nodes, pods could not get IP addresses because Multus could not reach the Kubernetes API, and further reboots spread the failure to every node
    Fyra Labs2023
  2. IL-0059 Single-master Kubernetes control plane kept dying because the kubelet restarted a healthy etcd on a failing kubeadm liveness probe
    NRE Labs2018

Certificate or credential expiry1 case

Component
GKE control planeKubernetes API serverNode poolRBAC and service accounts
Contributing factor
Limited debug accessSingle point of failure
Organisation
Webhook Relay
  1. IL-0113 The managed GKE control plane of Webhook Relay's main cluster failed because its API server certificates had never been rotated, and a manual credential rotation reported success without taking effect
    Webhook Relay2022

DNS or network fault1 case

Component
Container memory limitContainer registry and its credentials
Contributing factor
Harmful manual actionOverlapping failuresSingle point of failure
Organisation
GitHub
  1. IL-0199 A non-essential container exceeding its memory limit made GitHub's production pods unavailable, and their restarts failed because a DNS maintenance change had left the clusters unable to reach the image registry
    GitHub2020