Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases22 cases

22 to 22 of 22

  1. IL-0467 Boskos crash-looped after a node drain started it from a buggy StatefulSet image, failing Kubernetes PR jobs for 9 hours
    Kubernetes project2019
    Symptoms

    Overnight into 26 March 2019 (PDT), every Prow test job that needed a GCP project from Boskos, the Kubernetes test-infra service that hands out GCP projects to test jobs, failed. The required job pull-kubernetes-e2e-gce failed on every run, so no pull request could be merged in kubernetes/kubernetes for just over nine hours, and other projects saw similar failures.

    Container crash loopService outage
    Trigger

    At 23:04 PDT the node running the only Boskos pod was drained to be replaced with a 1.13 node during the upgrade of the Prow clusters from Kubernetes 1.11 to 1.13. The replacement pod started from the StatefulSet's current spec, which named a newer Boskos image containing the bug.

    Root cause
    Faulty deployment
    Component
    CI systemNode pool
    Contributing factor
    Change with a delayed effectUnverified assumptionSingle point of failureNo alert on limitChange started before off-hours
    Detection

    Users reported the failure on GitHub at 02:50 and on Slack at 05:16 and 06:27, and an engineer saw the Slack reports on waking at 07:16. The Boskos monitoring dashboard showed the failure, but no alert caught it.

    Remediation
    • Built and deployed a new Boskos image with the bug fixed
    • Cleaned up the GCP projects that had been left open
    Action items
    • Change the Boskos configuration so that deployments actually take effect (#11956)
    • Automate Boskos deploys (#11957)
    • Improve alerting on Boskos status (#11953)
    Lessons

    The write-up lists what went poorly: a ten-hour upgrade was started shortly after 17:00 when on-call coverage ended at 18:00; deployments were not checked to confirm that something had actually rolled out; the Boskos StatefulSet was not set up to perform updates, so changing its spec left the old pod running; the December change appears never to have been tested; and Boskos was deployed by hand, unlike most of Prow, which left more room for error. Once people were available, the problem was fixed quickly.

    View case
Check your cluster for this

These 22 cases involve Ingress controller and Cloud load balancer. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause6

Configuration change10 cases

Component
cert-manager2Cilium2Cloud load balancer2conntrack table2Ingress controller2Linux kernel2CDN and edge network1Configuration management tool1Configuration templating1ConfigMaps and Secrets1CoreDNS1Client-side DNS resolver and cache1Authoritative DNS servers and records1etcd1external-dns1HAProxy1Helm1HTTP and RPC client library1Karpenter1Kubernetes API server1kubelet1Node operating system image1Node pool1
Contributing factor
Change with a delayed effect10Unverified assumption3Retry storm3Unreviewed defaults2Monitoring blind spot2No configuration tests2Excessive blast radius1Configuration not in code1Ignored early signal1Unknown limit1Limited debug access1Harmful manual action1No alert on limit1No runbook1Shared credential1
Organisation
Wikimedia Foundation2Adyen1Datadog1DigitalOcean1LINE Plus1MIT Open Learning1Pusher1Python Software Foundation1Stytch1
  1. IL-0246 New pods on Adyen's big data Kubernetes nodes could not start, failing with "Cilium API client timeout exceeded"
    Adyen2026
  2. IL-0512 xpro.mit.edu returned 503 for an hour when its Kubernetes/APISIX origin certificate expired after renewal failed
    MIT Open Learning2026
  3. IL-0153 DOKS worker nodes went NotReady at 06:00 UTC on three days because a systemd timer in the Debian image ran apt upgrade and killed it mid kernel upgrade, breaking node networking
    DigitalOcean2023
  4. IL-0459 MediaWiki on Kubernetes terminated TLS with a wrong certificate after a deployment, and 1% of requests failed
    Wikimedia Foundation2023
  5. IL-0231 Deleting EKS managed node groups also deleted Karpenter's instance profile, and every Stytch worker node shut down
    Stytch2023
  6. IL-0513 PyPI backends on Kubernetes were unreachable over HTTPS for 20 minutes when the Ingress ELB lost its HTTPS listener
    Python Software Foundation2022
  7. IL-0177 WebSocket connections to LINE's messaging-hub on Kubernetes dropped all at once whenever another service's change reloaded a shared HAProxy L7 load balancer, and the reconnect wave overloaded a token API and filled nf_conntrack on Nginx
    LINE Plus2022
  8. IL-0461 Every rollout of Datadog's metrics query service on Kubernetes raised errors that the logs blamed on DNS
    Datadog2022
  9. IL-0458 Toolforge Kubernetes API server failed to start after a restart, logging "etcd cluster is unavailable or misconfigured"
    Wikimedia Foundation2019
  10. IL-0066 Broken ConfigMap change lay dormant for weeks, then took down Pusher's edge proxy when a cluster upgrade replaced its pods
    Pusher2019

Capacity exhaustion2 cases

Component
Jobs and CronJobs2Calico1Database1Helm1
Contributing factor
Change with a delayed effect2Unreviewed defaults1No runbook1
Organisation
CyberAgent1Wikimedia Foundation1
  1. IL-0435 Error rates rose as Aurora MySQL killed connections and declined queries to avoid OOM while load metrics stayed flat
    CyberAgent2026
  2. IL-0232 Calico pods in the codfw WikiKube cluster hit their resource limits and crash-looped, taking nodes out one by one
    Wikimedia Foundation2024

Breaking change in an upgrade5 cases

Component
Ingress controller3cert-manager2Cloud load balancer2Node operating system image2Admission webhook1Cilium1ConfigMaps and Secrets1Helm1Node pool1systemd-networkd1
Contributing factor
Change with a delayed effect5Monitoring blind spot2Automation amplification1Unreviewed defaults1No staged rollout1Ineffective rollback1Unrepresentative staging1
Organisation
2i2c1Datadog1Enigmo1Honeycomb1PortSwigger1
  1. IL-0098 After 2i2c moved from ingress-nginx to nginx-ingress, a hub's Ingress took priority over cert-manager's temporary Ingress for the same host, so new hubs could not get TLS certificates
    2i2c2026
  2. IL-0405 An expired AWS Load Balancer Controller webhook certificate left no healthy targets and took BSEE SaaS down
    PortSwigger2026
  3. IL-0203 A routine Kubernetes cluster system update made the AWS Load Balancer Controller fail, so new pods were never registered with the ALB and Honeycomb's EU1 ingest and UI went down once the last old pods were replaced
    Honeycomb2024
  4. IL-0198 An automatic Ubuntu security update restarted systemd-networkd, which flushed the IP rules Cilium relied on and cut more than 60 percent of Datadog's Kubernetes nodes off the network across regions and cloud providers
    Datadog2023
  5. IL-0449 Pods on Enigmo's new EKS managed node groups kept restarting or stuck Terminating and the nodes became unusable
    Enigmo2020

Certificate or credential expiry1 case

Component
Cluster autoscalerManaged Kubernetes control plane and provider API
Contributing factor
Change with a delayed effect
Organisation
Linode
  1. IL-0156 Revoking old certificates after a rotation left a certificate key mismatch between API endpoints and the cluster controller, and LKE cluster management and autoscaling in Atlanta returned 5xx errors
    Linode2025

Software defect3 cases

Component
Admission webhook1Cloud load balancer1GPU driver1Horizontal pod autoscaler1Ingress controller1Kubernetes API server1Linux kernel1
Contributing factor
Change with a delayed effect3Ignored early signal2Monitoring blind spot2Excessive blast radius1No configuration tests1Single point of failure1
Organisation
Airbnb1Barush Mendez1DFDS1
  1. IL-0138 A failed Nvidia driver build left broken kernel packages on the only Kubernetes master node, which hit a kernel panic on reboot after a power outage and took the homelab cluster down for six days
    Barush Mendez2025
  2. IL-0106 A malformed Traefik IngressRoute triggered a Traefik v2 bug once a node rollover restarted every Traefik pod, and all Kubernetes services returned 404
    DFDS2021
  3. IL-0032 Mutating admission controller generated broken JSON patches, silently blocking deploys for a week until a production service ran down to two pods
    Airbnb2020

Faulty deployment1 case

Component
CI systemNode pool
Contributing factor
Unverified assumptionChange started before off-hoursChange with a delayed effectNo alert on limitSingle point of failure
Organisation
Kubernetes project
  1. IL-0467 Boskos crash-looped after a node drain started it from a buggy StatefulSet image, failing Kubernetes PR jobs for 9 hours
    Kubernetes project2019