Incident Library
Applied filters FactorUnverified assumptionClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases35 cases

26 to 30 of 35

  1. IL-0062 Intermittent 502 errors behind a GKE Ingress because pods shut down before the load balancer stopped sending them traffic
    Ravelin2019
    Symptoms

    After Ravelin moved its API layer from VMs into its GKE cluster behind a Kubernetes Ingress, the integration tests occasionally received 502 errors.

    Intermittent connection failuresStale endpoint routing
    Trigger

    Pods behind the Ingress being removed by their controller, as happens during rolling deploys and scaling.

    Root cause
    Software defect
    Component
    Ingress controllerPod termination
    Contributing factor
    Unverified assumption
    Detection

    Integration tests occasionally received 502 errors.

    Lessons

    Removing a pod's endpoint and signalling the pod happen at the same time, and an ingress can take a long time to catch up. A pod behind an ingress should keep serving after SIGTERM until its grace period ends, and the grace period should be longer than the load balancer's reprogramming time.

    View case
  2. IL-0061 A Terraform pull request merged out of order deleted a production GKE cluster, and recreating it deleted a second one
    Spotify2019
    Symptoms

    Terraform, applied from the shared cluster configuration repository after a merge, deleted Spotify's production GKE cluster in Asia. While the team tried to recreate it, Terraform deleted a second production cluster, so two of the three production clusters were gone.

    Resources deleted
    Trigger

    A feature team's pull request that added three users to an existing RBAC role binding file was merged before the cluster operators' pull request that imported the Asia cluster into Terraform state. The review build of the import pull request had already written the cluster into the shared state file, and the merged configuration had no definition of the Asia cluster, so Terraform removed it.

    Root cause
    Configuration change
    Component
    TerraformCI systemGKE control plane
    Contributing factor
    No safeguard on destructive changeMissed in reviewUnverified assumption
    Detection

    Internal service discovery could not reach the deleted cluster and the cluster operator team was paged.

    Remediation
    • removed the deleted clusters from the service discovery configuration and restarted service discovery so clients fell back to the instances outside Kubernetes
    • one team that ran half on Kubernetes and half on VMs created more VMs outside Kubernetes to handle the load
    • recreated the clusters, using the hourly backups
    • updated the places where the old clusters' master IPs were hardcoded
    Action items
    • Post the Terraform dry-run output as a comment on each pull request
    • Require a status check that the branch is up to date with master, so the dry run reflects what will actually happen
    • Require an approving review on pull requests
    • Fail reviews automatically when certain keywords such as destroy appear; cluster deletions are done manually outside this repository
    • Add linters and validators for the cluster configuration
    • Schedule and announce disaster recovery tests that cover a cluster going away and smaller failures
    Lessons

    Plan for failure from the start and migrate large infrastructure gradually, so there is redundancy and a way back. A backup whose restore has never been tested is not a backup. Practising disaster recovery paid off: restoring the first deleted cluster took three and a quarter hours, and the team can now restore a larger cluster in one hour. What matters is not who made the mistake but what was learned from it.

    View case
  3. IL-0066 Broken ConfigMap change lay dormant for weeks, then took down Pusher's edge proxy when a cluster upgrade replaced its pods
    Pusher2019
    Symptoms

    During a cluster upgrade, all traffic to Pusher's products stopped. The speaker recalls that requests were failing with 500 errors.

    Service outageHigh error rate
    Trigger

    A cluster upgrade replaced nodes, so the bridge pods restarted and loaded a ConfigMap that had been changed three or four weeks earlier.

    Root cause
    Configuration change
    Component
    ConfigMaps and Secrets
    Contributing factor
    Change with a delayed effectUnverified assumption
    Action items
    • Build and run Wave so that changes to ConfigMaps and Secrets roll out new pods immediately
    Lessons

    A ConfigMap change does not restart the pods that use it, so for an application that cannot reload its configuration the change takes effect only when pods are next replaced, possibly weeks later during unrelated work. Tie configuration to the pod template so a bad change fails at the moment it is made.

    View case
  4. IL-0011 HAProxy ingress piled requests onto the same pods because a Go template visited the backend map in sorted pod IP order
    Skyscanner2019
    Symptoms

    On a Sunday, a team using one of Skyscanner's Kubernetes clusters manually paged the out-of-hours Kubernetes support because their service showed elevated latency, worst at the p90 and maximum levels.

    High latencyLoad imbalanceOOMKilledIntermittent connection failures
    Trigger

    Upgrading the HAProxy ingress controller to v0.6 and going back from Service virtual IPs to individual pod IPs as backends, on the understanding that the sort-backends flag introduced in v0.5, false by default, meant backends were randomly shuffled.

    Root cause
    Software defect
    Component
    Ingress controllerHAProxyConfiguration templating
    Contributing factor
    Unrepresentative stagingUnverified assumptionIgnored early signal
    Detection

    A cluster user manually paged the out-of-hours Kubernetes support about elevated latency; Prometheus queries then showed the uneven request distribution across pods.

    Remediation
    • switched the affected service from pod IPs back to the Service virtual IP through a config change in the internal deployment tool Slingshot
    • patched the controller to keep each backend's servers in an insertion-ordered slice and changed the template to iterate over that slice
    • tested the patch in sandbox clusters and rolled it out to the production clusters
    Action items
    • Run the internal controller patch in production and return to pod IPs as the default backends
    • Raise the problem upstream with a GitHub issue and pull request
    • Test changes to cluster components against the specific behaviour and conditions that matter
    Lessons

    The team already knew that sorted backends cause hot-podding and still did not test the new flag well enough to see that it did not work; changes to cluster components need tests aimed at the exact behaviour that matters, under conditions like production. A keyword can behave differently in different parts of a language: range over a map is unordered in Go code but sorted in text/template. A small cluster component can do outsized harm to every user of the cluster and, in the end, to customers.

    View case
  5. IL-0054 All ingress traffic lost while draining a GKE node pool because the load balancer only targeted new nodes without ingress-nginx pods
    DevOps Hof2019
    Symptoms

    While the old nodes of a GKE cluster were being drained to move workloads to a newly created node pool, none of the services in the cluster could be reached through ingress-nginx any more.

    Service outageProbe failures
    Trigger

    Cordoning and draining the old node pool removed those nodes from the GCP load balancer backend while the ingress-nginx pods were still running on them.

    Root cause
    Operator error
    Component
    Ingress controllerkube-proxyNode pool
    Contributing factor
    Unreviewed defaultsUnverified assumption
    Detection

    The team noticed during the drain that no service in the cluster was reachable through ingress-nginx.

    Remediation
    • deleted the nginx-ingress-controller pods still running on the old nodes so they were rescheduled on the new, ready nodes
    Lessons

    With externalTrafficPolicy: Local, the load balancer can only reach nodes that run an ingress pod, and cordoned nodes leave the load balancer backend immediately; if the ingress pods stay behind on cordoned nodes, all ingress traffic stops. Before any change that might touch traffic, look twice into the documentation.

    View case
Check your cluster for this

These 35 cases involve Telemetry agent and Application database. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause10

Configuration change9 cases

Component
Cloud load balancer3Terraform2Admission webhook1In-memory cache1CI system1Configuration templating1ConfigMaps and Secrets1conntrack table1Container runtime1Application database1GKE control plane1HAProxy1Horizontal pod autoscaler1Ingress controller1Kubernetes API server1Application logging library1Container memory limit1Persistent volumes and stateful sets1Telemetry agent1AWS VPC CNI1
Contributing factor
Unverified assumption9Excessive blast radius3Change with a delayed effect2No configuration tests2No safeguard on destructive change2Retry storm2Monitoring blind spot1No health probes1No headroom1No staged rollout1Missed in review1Single point of failure1
Organisation
DFDS2AdGuard VPN1GitLab1Grafana Labs1LINE+1MUSINSA1Pusher1Spotify1
  1. IL-0179 With DEBUG logging and synchronous log output, a high-traffic service filled the 64 KB stdout pipe, so request threads blocked in write() while holding database connections and HikariCP ran dry, failing requests in 30-second bursts
    MUSINSA2026
  2. IL-0063 VPN outage after recreating a Redis StatefulSet restarted the whole Redis cluster without readiness checks
    AdGuard VPN2025
  3. IL-0192 A Crossplane composition merge-policy change for a TLS policy update switched Kubernetes Services from LoadBalancer to ClusterIP and deleted the load balancers behind about 25% of Grafana Cloud services
    Grafana Labs2025
  4. IL-0177 WebSocket connections to LINE's messaging-hub on Kubernetes dropped all at once whenever another service's change reloaded a shared HAProxy L7 load balancer, and the reconnect wave overloaded a token API and filled nf_conntrack on Nginx
    LINE+2022
  5. IL-0209 A too-high HPA CPU target kept GitLab.com web pods from scaling up as traffic moved from VMs to Kubernetes, breaching the web latency SLO for 15 minutes
    GitLab2021
  6. IL-0105 Replacing the AWS VPC CNI during a Terraform refactoring detached the ALB target groups, and every service in the Kubernetes cluster returned 503
    DFDS2021
  7. IL-0108 An Alcide admission controller applied to the production cluster during a workshop made create and update requests on Deployments time out
    DFDS2020
  8. IL-0061 A Terraform pull request merged out of order deleted a production GKE cluster, and recreating it deleted a second one
    Spotify2019
  9. IL-0066 Broken ConfigMap change lay dormant for weeks, then took down Pusher's edge proxy when a cluster upgrade replaced its pods
    Pusher2019

Software defect8 cases

Component
Ingress controller2Node operating system image2Argo CD1Configuration templating1Container CPU limit1Application database1Client-side DNS resolver and cache1GPU driver1HAProxy1Karpenter1Linux kernel1Container memory limit1Persistent volumes and stateful sets1Pod termination1Telemetry agent1
Contributing factor
Unverified assumption8Automation amplification1Unreviewed defaults1Ignored early signal1Unpatched known bug1Harmful manual action1No safeguard on destructive change1No staged rollout1Single point of failure1Unrepresentative staging1
Organisation
Argo CD users1Buffer1Inflab1Palark1Ravelin1Skyscanner1thredUP1Toyota Connected1
  1. IL-0058 Argo CD deleted namespaces shared by several applications, leaving about 97 applications stuck Missing and OutOfSync
    Argo CD users2026
  2. IL-0171 Karpenter node auto-repair replaced EKS GPU nodes about 10 minutes after they became Ready, in a loop, because the Bottlerocket NVIDIA AMI lacked libdcgm.so and set AcceleratedHardwareReady to False
    Inflab2025
  3. IL-0029 Slow user-facing services from CPU throttling on containers that used far less CPU than their limit
    Buffer2020
  4. IL-0196 Adding a second Redis replica and lowering memory limits in one RedisFailover change restarted the only Redis pod that held data, and Sentinel promoted the empty new replica, erasing the data
    Palark2020
  5. IL-0062 Intermittent 502 errors behind a GKE Ingress because pods shut down before the load balancer stopped sending them traffic
    Ravelin2019
  6. IL-0011 HAProxy ingress piled requests onto the same pods because a Go template visited the backend map in sorted pod IP order
    Skyscanner2019
  7. IL-0050 Pods on Kubernetes failed to resolve an external Redis hostname because of ndots:5, the musl resolver in Alpine images, and Netty's DNS resolver
    Toyota Connected2018
  8. IL-0051 A third-party log shipping client leaked file descriptors with every new container until nodes failed with too many open files
    thredUP2018

Breaking change in an upgrade1 case

Component
Cluster autoscaler
Contributing factor
Unverified assumptionDeferred upgrades
Organisation
2i2c
  1. IL-0097 A cluster autoscaler version bump was incompatible with the one cluster still on Kubernetes 1.33, so no scale-up events fired and new nodes were not created
    2i2c2026

Faulty deployment3 cases

Component
Telemetry agent2Cloud load balancer1GPU driver1Ingress controller1Pod termination1Service discovery1
Contributing factor
Unverified assumption3Excessive blast radius1Unreviewed defaults1Monitoring blind spot1Missed in review1
Organisation
Airbnb1Linode1Yonghyun Kim1
  1. IL-0160 A repaired legacy GPU monitoring script sent a firmware query that triggers a race condition in the GPU System Processor of NVIDIA RTX 4000 Ada GPUs, locking up GPU Linodes and LKE worker nodes in several regions
    Linode2026
  2. IL-0188 Rolling updates on EKS returned 502 errors for about 30 seconds because Kubernetes deleted old pods as soon as new pods passed readiness, before the ALB had marked the new pod targets healthy
    Yonghyun Kim2025
  3. IL-0034 A crashing stats agent sidecar marked every pod unready and the service mesh pulled all services out of traffic
    Airbnb2019

Capacity exhaustion6 cases

Component
Pod scheduling and placement2Cluster autoscaler1Application database1etcd1Jobs and CronJobs1Kubernetes API server1Linux kernel1Container memory limit1Node pool1Shared network filesystem1VPC subnet or pod address range1Telemetry agent1Training data loader1AWS VPC CNI1
Contributing factor
Unverified assumption6Automation amplification2Excessive blast radius2No capacity planning2Unreviewed defaults1Unknown limit1Harmful manual action1Monitoring blind spot1No resource requests or limits1Overlapping failures1Retry storm1Ineffective rollback1Single point of failure1Unrepresentative staging1
Organisation
2i2c1Civis Analytics1Hugging Face1Moonlight1Neon1Universe1
  1. IL-0129 Throughput of the 384-GPU SmolLM3 pretraining run collapsed because shared WekaFS storage evicted dataset shards to S3, then kept dropping on local NVMe because the nanotron dataloader built an index that grew with the step count
    Hugging Face2025
  2. IL-0006 VPC subnets ran out of IP addresses because the AWS VPC CNI kept IPs on full nodes, and WARM_IP_TARGET changes then blocked pod starts after every pod deletion
    Neon2025
  3. IL-0094 Student pods filled the shared nodes of a JupyterHub cluster and starved the hub's core pods, freezing the hub
    2i2c2020
  4. IL-0064 CPU-heavy pods packed onto one Google Cloud node drove it to kernel panics, and the crash moved from node to node until the site was down
    Moonlight2019
  5. IL-0075 Kubernetes API servers pegged on CPU when the cluster grew to about 290 nodes, traced to Datadog agents polling the API
    Civis Analytics2019
  6. IL-0036 A reindex Job that kept failing exhausted one node after another until the whole cluster was down
    Universe2017

Dependency failure1 case

Component
GPU driverInfiniBand fabricNCCLPyTorch
Contributing factor
Unverified assumptionMonitoring blind spotPartial rollout
Organisation
Imbue
  1. IL-0025 Failed boots, InfiniBand link flapping, NCCL hangs and throughput sags while bringing up a 4,088-GPU H100 training cluster
    Imbue2024

DNS or network fault2 cases

Component
CoreDNS1Client-side DNS resolver and cache1Telemetry agent1AWS VPC CNI1
Contributing factor
Unverified assumption2Unreviewed defaults2Ignored early signal1Unknown limit1
Organisation
Adevinta1MindTickle1
  1. IL-0083 Intermittent 5xx errors in a multi-tenant Kubernetes cluster came from internal DNS: the node-local dnsmasq hit its 150 concurrent query limit, did not cache cluster.local answers, and ndots search lookups flooded CoreDNS
    Adevinta2023
  2. IL-0056 Calls from EKS pods to services behind an NLB in another VPC failed intermittently because the AWS VPC CNI did source NAT on the node
    MindTickle2020

unknown1 case

Component
CalicoContainer runtimeMultus meta-CNI
Contributing factor
Unverified assumptionHarmful manual action
Organisation
Fyra Labs
  1. IL-0087 After a cold reboot of RKE2 nodes, pods could not get IP addresses because Multus could not reach the Kubernetes API, and further reboots spread the failure to every node
    Fyra Labs2023

Operator error2 cases

Component
Elasticsearch or OpenSearch1Helm1Ingress controller1kube-proxy1Node pool1Persistent volumes and stateful sets1
Contributing factor
Unverified assumption2Unreviewed defaults1
Organisation
DevOps Hof1Palark1
  1. IL-0197 Moving a three-node Elasticsearch StatefulSet to new persistent volumes split the cluster in two, because the Helm chart hardcoded cluster.initial_master_nodes to elasticsearch-0
    Palark2021
  2. IL-0054 All ingress traffic lost while draining a GKE node pool because the load balancer only targeted new nodes without ingress-nginx pods
    DevOps Hof2019

Cloud provider outage2 cases

Component
Application database2Helm1Node root disk1Persistent volumes and stateful sets1
Contributing factor
Unverified assumption2Excessive blast radius1Unpatched known bug1Overlapping failures1
Organisation
Devsisters1Grafana Labs1
  1. IL-0172 An AWS Tokyo data centre cooling failure took down 6 of 60 CockroachDB nodes on Kubernetes within 22 minutes, and ranges lost quorum because the Helm-deployed cluster had no availability zone locality configured
    Devsisters2021
  2. IL-0193 A GCP persistent disk degradation took down a Cassandra StatefulSet whose OrderedReady policy blocked recovery, and Metrictank could not reconnect to the recreated pods, causing a 23-hour Graphite query outage
    Grafana Labs2019