Incident Library
Applied filters FactorUnrepresentative stagingClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases14 cases

13 to 14 of 14

  1. IL-0011 HAProxy ingress piled requests onto the same pods because a Go template visited the backend map in sorted pod IP order
    Skyscanner2019
    Symptoms

    On a Sunday, a team using one of Skyscanner's Kubernetes clusters manually paged the out-of-hours Kubernetes support because their service showed elevated latency, worst at the p90 and maximum levels.

    High latencyLoad imbalanceOOMKilledIntermittent connection failures
    Trigger

    Upgrading the HAProxy ingress controller to v0.6 and going back from Service virtual IPs to individual pod IPs as backends, on the understanding that the sort-backends flag introduced in v0.5, false by default, meant backends were randomly shuffled.

    Root cause
    Software defect
    Component
    Ingress controllerHAProxyConfiguration templating
    Contributing factor
    Unrepresentative stagingUnverified assumptionIgnored early signal
    Detection

    A cluster user manually paged the out-of-hours Kubernetes support about elevated latency; Prometheus queries then showed the uneven request distribution across pods.

    Remediation
    • switched the affected service from pod IPs back to the Service virtual IP through a config change in the internal deployment tool Slingshot
    • patched the controller to keep each backend's servers in an insertion-ordered slice and changed the template to iterate over that slice
    • tested the patch in sandbox clusters and rolled it out to the production clusters
    Action items
    • Run the internal controller patch in production and return to pod IPs as the default backends
    • Raise the problem upstream with a GitHub issue and pull request
    • Test changes to cluster components against the specific behaviour and conditions that matter
    Lessons

    The team already knew that sorted backends cause hot-podding and still did not test the new flag well enough to see that it did not work; changes to cluster components need tests aimed at the exact behaviour that matters, under conditions like production. A keyword can behave differently in different parts of a language: range over a map is unordered in Go code but sorted in text/template. A small cluster component can do outsized harm to every user of the cluster and, in the end, to customers.

    View case
  2. IL-0014 A GKE upgrade to a withdrawn version stalled on slow drains and a stuck node pool, then broke an ingress and confused the HPA
    loveholidays2019
    Symptoms

    On the evening of a long-overdue upgrade of a loveholidays production GKE cluster from 1.10.X to 1.12.5-gke.10, planned to take 2 to 2.5 hours, the master upgrade and the first two 7-node NodePools went smoothly. After that the NodePool upgrades slowed to a crawl. Each node is cordoned, drained and upgraded in turn. Even a 14-node cluster at about 5 minutes per node took over an hour, and with about 100 more nodes to go, some of them taking over 15 minutes to drain, the sequential drains were too slow to finish at a reasonable hour.

    Upgrade stallPods PendingScale-up failureService outageAutoscaler misbehaviour
    Trigger

    A manual upgrade of the GKE masters and NodePools from 1.10.X to 1.12.5-gke.10, two versions at once, on 19 March 2019; GKE's release notes of 14 March had already said that version was no longer available for new clusters or master upgrades because masters showed elevated error rates, but it could still be selected.

    Root cause
    Cloud provider outage
    Component
    GKE control planeNode poolCluster autoscalerIngress controllerHorizontal pod autoscaler
    Contributing factor
    Unrepresentative stagingOverlapping failuresIgnored early signal
    Detection

    The slow drains, zombie nodes and stuck NodePool were seen by the engineer running the upgrade; the broken Ingress was noticed after the upgrade and explained by Google support; the HPA problem was noticed the next day.

    Remediation
    • drained the remaining NodePool nodes ahead of the upgrade with kubectl drain --ignore-daemonsets --delete-local-data
    • deleted the zombie instances from the managed instance group with gcloud compute instance-groups managed delete-instances
    • cancelled the stuck UPGRADE_NODES operation with gcloud beta container operations cancel
    • used kubectl port-forward in place of the broken internal Ingresses
    • scaled down the older of two active ReplicaSets, or triggered a dummy deployment, to bring the HPA back
    Action items
    • Alert when a HorizontalPodAutoscaler is at 70% and at 100% of its capacity
    • Alert when a deployment has had two active ReplicaSets for 30 minutes and for 60 minutes
    • Re-evaluate termination grace periods that were set to arbitrary large values
    • Read the GKE release notes before every cluster upgrade
    • Use auto-upgrade in non-production environments, and manual cluster upgrades with automatic node upgrades in production
    • Keep environments as identical as possible, including resource names
    Lessons

    Read the release notes before upgrading: the target version had been withdrawn five days earlier. Drain time depends on pod configuration, so emptyDir volumes, long grace periods and missing disruption budgets turn a NodePool upgrade into hours. Pre-production only catches problems like the ingress bug if it matches production down to resource names. Auto-upgrades in non-production surface problems early, and a mix of manual cluster upgrades and automatic node upgrades in production keeps clusters current without much effort.

    View case
Check your cluster for this

These 14 cases involve Ingress controller and Node pool. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause5

Capacity exhaustion3 cases

Component
Shared network filesystem2In-memory cache1Application database1Node pool1Pod scheduling and placement1Training data loader1
Contributing factor
Unrepresentative staging3Unverified assumption1Monitoring blind spot1No alert on limit1No staged rollout1Overlapping failures1Retry storm1Ineffective rollback1Single point of failure1
Organisation
Hugging Face1Lablup Inc1PostHog1
  1. IL-0137 Training initialisation on a 60-node NVIDIA B200 cluster took more than 8 hours instead of minutes because fragmented small random I/O from all nodes saturated the metadata service of the VAST Data NFS storage
    Lablup Inc2026
  2. IL-0129 Throughput of the 384-GPU SmolLM3 pretraining run collapsed because shared WekaFS storage evicted dataset shards to S3, then kept dropping on local NVMe because the nanotron dataloader built an index that grew with the step count
    Hugging Face2025
  3. IL-0100 PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queries
    PostHog2025

Breaking change in an upgrade4 cases

Component
Ingress controller3Admission webhook2Calico1Cloud load balancer1etcd1Kubernetes API server1Web application firewall rules1
Contributing factor
Unrepresentative staging4Monitoring blind spot2Untested backup1Configuration not in code1Change with a delayed effect1Single point of failure1
Organisation
UK Ministry of Justice2Honeycomb1Reddit1
  1. IL-0092 Upgrading ingress-nginx from 1.12.0 to 1.14.3 brought a newer ModSecurity and OWASP Core Rule Set whose false positives blocked HMPPS Auth, so users could not sign in to their applications
    UK Ministry of Justice2026
  2. IL-0203 A routine Kubernetes cluster system update made the AWS Load Balancer Controller fail, so new pods were never registered with the ALB and Honeycomb's EU1 ingest and UI went down once the last old pods were replaced
    Honeycomb2024
  3. IL-0060 Kubernetes 1.24 upgrade removed the master node label that Calico route reflectors selected on, dropping all pod routes and taking Reddit down for 314 minutes
    Reddit2023
  4. IL-0090 After an upgrade of the ModSec ingress controller module, its validating webhook served a certificate for the wrong service name and blocked every Ingress create and change
    UK Ministry of Justice2021

Software defect2 cases

Component
Configuration templating1HAProxy1Ingress controller1Node pool1Terraform1
Contributing factor
Unrepresentative staging2Unverified assumption1Ignored early signal1Deferred upgrades1
Organisation
Skyscanner1ZenML1
  1. IL-0078 Enabling EKS secrets encryption with KMS through Terraform removed worker node IAM permissions and took down all ZenML Pro workspaces
    ZenML2025
  2. IL-0011 HAProxy ingress piled requests onto the same pods because a Go template visited the backend map in sorted pod IP order
    Skyscanner2019

Configuration change4 cases

Component
Telemetry agent2CNI plugin1Configuration templating1Container runtime1Client-side DNS resolver and cache1Kubernetes API server1Node launch template1Container memory limit1Node pool1Service discovery1Terraform1
Contributing factor
Unrepresentative staging4Unreviewed defaults2Monitoring blind spot2Automation amplification1Ignored early signal1Cache-masked failure1Limited debug access1Harmful manual action1No configuration tests1No runbook1No staged rollout1
Organisation
OpenAI2DFDS1Grafana Labs1
  1. IL-0005 A new telemetry service overloaded the Kubernetes API servers of large clusters and broke DNS-based service discovery for all OpenAI services
    OpenAI2024
  2. IL-0109 A retried node rollover on a DFDS EKS cluster set the Auto Scaling group back to 15 instances, and the default termination policy removed every healthy worker node, leaving the cluster with no nodes
    DFDS2023
  3. IL-0116 A CNI configuration change incompatible with a hardware variant that exists only in production cut network connectivity in an OpenAI GPU cluster and took text-davinci-003 fully down
    OpenAI2023
  4. IL-0191 A label selector accidentally added to the Loki distributor Service matched only the new pods of a rollout, which took all write traffic, were OOM-killed and stalled the rollout
    Grafana Labs2022

Cloud provider outage1 case

Component
Cluster autoscalerGKE control planeHorizontal pod autoscalerIngress controllerNode pool
Contributing factor
Ignored early signalOverlapping failuresUnrepresentative staging
Organisation
loveholidays
  1. IL-0014 A GKE upgrade to a withdrawn version stalled on slow drains and a stuck node pool, then broke an ingress and confused the HPA
    loveholidays2019