Incident Library
Applied filters Root causeFaulty deploymentDomainKubernetesClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases20 cases

16 to 20 of 20

  1. IL-0088 A validating admission webhook added by a new ingress controller module version made the live-1 cluster reject every new Ingress with an x509 error
    UK Ministry of Justice2020
    Symptoms

    At 12:39 on 7 September 2020 the Cloud Platform team found that the Ingress API of the live-1 cluster was refusing every POST request. A user who provisioned a new service could not create an ingress into the cluster. The incident was declared at 12:54.

    Deploys failing
    Trigger

    Version 0.1.0 of the team's ingress controller module started creating a ValidatingWebhookConfiguration resource, and one new 0.1.0 ingress controller in the live-1 cluster failed to be created because of AWS resource limits.

    Root cause
    Faulty deployment
    Component
    Admission webhookIngress controller
    Contributing factor
    Single point of failure
    Detection

    The Ingress API was found refusing 100% of POST requests at 12:39; the incident was declared at 12:54.

    Remediation
    • removed by hand all the additional admission controllers created by module version 0.1.0
    • removed the admission webhook from the ingress controller module and released version 0.1.1
    • upgraded every ingress module on version 0.1.0 to 0.1.1
    Lessons

    Turning on the validating webhook in the shared ingress controller module made the webhook a single point of failure for every ingress controller pod in the ingress-controllers namespace: when one controller could not come up, no team could create an ingress.

    View case
  2. IL-0034 A crashing stats agent sidecar marked every pod unready and the service mesh pulled all services out of traffic
    Airbnb2019
    Symptoms

    An operation to upgrade the stats agent in Kubernetes was announced in advance, with a warning that metrics might be unavailable to services for a few minutes. A few minutes later a reply came in saying that all of the sender's Kubernetes services seemed to be down.

    Probe failuresService outage
    Trigger

    A scheduled upgrade of the stats agent sidecar, during which the agent container crashed.

    Root cause
    Faulty deployment
    Component
    Telemetry agentService discovery
    Contributing factor
    Unverified assumptionExcessive blast radius
    Detection

    A reply to the operation announcement, a few minutes later, said that all of the sender's Kubernetes services appeared to be down.

    Action items
    • Mark logging, tracing and stats agent containers as non-critical and leave them out of the service mesh's pod readiness check
    Lessons

    Every container in a pod affects the pod's readiness, so each sidecar needs a deliberate decision about how its health should count. A sidecar that crashes is not ready even when it has no probes, and any logic that requires all containers to be ready will take the pod out of traffic.

    View case
  3. IL-0149 Upgrades of AKS clusters on Virtual Machine Scale Sets failed with a 'platform image is not available' error after a service update left a backend service unhealthy
    Microsoft Azure2019
    Symptoms

    From 07:20 UTC on 28 September 2019, customers running Azure Kubernetes Service clusters on Virtual Machine Scale Sets (VMSS) got failures when they tried any cluster upgrade. The error read: "The platform image 'microsoft-aks:aks:aks-ubuntu-1604-201909:2019.08.21' is not available. Verify that all fields in the storage profile are correct." The issue title calls the error PlatformImageNotFound.

    Upgrade stall
    Trigger

    A recent service update affected instances of a backend service, which became unhealthy and stopped requests from completing.

    Root cause
    Faulty deployment
    Component
    Node operating system imageNode pool
    Contributing factor
    –
    Detection

    Customers received failure messages when they tried to upgrade their clusters.

    Remediation
    • published az vmss update commands that set the scale set's image version to 2019.09.25 and SKU to aks-ubuntu-1604-201909, after which the upgrade could be run again
    • mitigated affected clusters by hand (AKS and VMSS teams)
    • rolled out hot fixes to all regions, completed by 15 October 2019
    View case
  4. IL-0467 Boskos crash-looped after a node drain started it from a buggy StatefulSet image, failing Kubernetes PR jobs for 9 hours
    Kubernetes project2019
    Symptoms

    Overnight into 26 March 2019 (PDT), every Prow test job that needed a GCP project from Boskos, the Kubernetes test-infra service that hands out GCP projects to test jobs, failed. The required job pull-kubernetes-e2e-gce failed on every run, so no pull request could be merged in kubernetes/kubernetes for just over nine hours, and other projects saw similar failures.

    Container crash loopService outage
    Trigger

    At 23:04 PDT the node running the only Boskos pod was drained to be replaced with a 1.13 node during the upgrade of the Prow clusters from Kubernetes 1.11 to 1.13. The replacement pod started from the StatefulSet's current spec, which named a newer Boskos image containing the bug.

    Root cause
    Faulty deployment
    Component
    CI systemNode pool
    Contributing factor
    Change with a delayed effectUnverified assumptionSingle point of failureNo alert on limitChange started before off-hours
    Detection

    Users reported the failure on GitHub at 02:50 and on Slack at 05:16 and 06:27, and an engineer saw the Slack reports on waking at 07:16. The Boskos monitoring dashboard showed the failure, but no alert caught it.

    Remediation
    • Built and deployed a new Boskos image with the bug fixed
    • Cleaned up the GCP projects that had been left open
    Action items
    • Change the Boskos configuration so that deployments actually take effect (#11956)
    • Automate Boskos deploys (#11957)
    • Improve alerting on Boskos status (#11953)
    Lessons

    The write-up lists what went poorly: a ten-hour upgrade was started shortly after 17:00 when on-call coverage ended at 18:00; deployments were not checked to confirm that something had actually rolled out; the Boskos StatefulSet was not set up to perform updates, so changing its spec left the old pod running; the December change appears never to have been tested; and Boskos was deployed by hand, unlike most of Prow, which left more room for error. Once people were available, the problem was fixed quickly.

    View case
  5. IL-0466 Prow scheduled test job pods onto its service cluster, leaving hundreds Pending and starving Prow's own services
    Kubernetes project2019
    Symptoms

    On 8 February 2019 the Kubernetes project's Prow instance at prow.k8s.io became very slow: its controllers took about five minutes to respond to the hook, bot commands such as the cat command stopped working, and the prow.k8s.io site was hard to reach. GitHub status was normal, and the Prow service nodes were found running above 100% of CPU and memory.

    Pods PendingSustained high node CPUHigh latency
    Trigger

    At 13:20 PST an auto-generated bump PR for Prow was merged and deployed, and Prow started scheduling workloads onto the service cluster.

    Root cause
    Faulty deployment
    Component
    CI system
    Contributing factor
    Unrepresentative stagingNo resource requests or limitsLimited debug access
    Detection

    While working on a PR, a maintainer noticed the bot's cat command failing, found GitHub status normal, learned on Slack that the Prow nodes were above 100% CPU and memory, and then found 600 pending pods in the test-pods namespace.

    Remediation
    • Rolled Prow back to v20190206-b01b288 and cleaned up the pods and ProwJobs in the service cluster
    • Applied the revert by hand with kubectl apply after the deployment permission was missing
    • Sent a fix, PR 11209, so the client no longer overwrites a provided literal default client
    • Rolled Prow back to v20190125-2aca69d, from before the bug was introduced
    • Cleaned up the wrongly scheduled pods and ProwJobs again and confirmed Prow was healthy
    Action items
    • Find out why the bug did not show deterministically
    • Be more careful with sweeping Prow changes, with more end-to-end tests or a real staging Prow
    Lessons

    The behaviour was not deterministic, and the change was deployed on a Friday afternoon. Prow instances are set up differently, for example in how build clusters are defined, so another project's Prow running a version fine does not show it is safe for the Kubernetes Prow, which needs a staging environment that looks like it.

    View case
Check your cluster for this

These 20 cases involve Node pool and CI system. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Faulty deployment20 cases

Component
Node pool7CI system4Ingress controller3Telemetry agent3Cloud load balancer2Cluster autoscaler2ConfigMaps and Secrets2Helm2Admission webhook1In-memory cache1Calico1Container CPU limit1Database1Descheduler1GPU driver1Model inference server1Managed Kubernetes control plane and provider API1Network control plane (SDN)1Node root disk1Node operating system image1Pod termination1Progressive delivery controller1Container registry and its credentials1Service discovery1
Contributing factor
Unverified assumption5Excessive blast radius4No alert on limit3Unreviewed defaults2Limited debug access2Monitoring blind spot2No staged rollout2Overlapping failures2Missed in review2Single point of failure2Unrepresentative staging2Gap in automated tests2Signal buried in noise1Automation amplification1Change started before off-hours1Change with a delayed effect1Unknown limit1Harmful manual action1No configuration tests1No resource requests or limits1No runbook1No headroom1Partial rollout1Ineffective rollback1Dependence on a warm cache1
Organisation
Linode5Google Cloud2Kubernetes project2UK Ministry of Justice2Airbnb1CircleCI1Gitpod1Gunosy1IONOS Cloud1Lucidworks1Microsoft Azure1Wikimedia Foundation1Yonghyun Kim1
  1. IL-0237 A Kubernetes patch upgrade evicted every Ray Serve head and worker pod at once, and Lucidworks hosted LLMs went down
    Lucidworks2026
  2. IL-0166 A compute host update removed the credential parser used to decode StackScripts, so new Linodes and every new LKE and LKE-E node failed to boot and autoscaling and node recycling failed in all regions
    Linode2026
  3. IL-0234 An onboarding operator deployed without its scope file installed dev applications on other IONOS Kubernetes clusters
    IONOS Cloud2026
  4. IL-0167 A phased rollout of an LKE software release in Osaka stalled LKE-E cluster creation, version upgrades and control plane ACL changes
    Linode2026
  5. IL-0160 A repaired legacy GPU monitoring script sent a firmware query that triggers a race condition in the GPU System Processor of NVIDIA RTX 4000 Ada GPUs, locking up GPU Linodes and LKE worker nodes in several regions
    Linode2026
  6. IL-0188 Rolling updates on EKS returned 502 errors for about 30 seconds because Kubernetes deleted old pods as soon as new pods passed readiness, before the ALB had marked the new pod targets healthy
    Yonghyun Kim2025
  7. IL-0020 During an EKS 1.30 upgrade the descheduler tried to reschedule every pod on the cordoned nodes at once and production workloads went down
    UK Ministry of Justice2025
  8. IL-0324 Three MediaWiki deployments lost Memcached after a PHP 8.1 rollout and overloaded the External Storage database cluster
    Wikimedia Foundation2025
  9. IL-0530 Virtual router controllers in us-central1 ran out of memory in a loop, cutting networking for new and migrated VMs
    Google Cloud2023
  10. IL-0142 GKE clusters on 1.24 and later wrote unexpected fluentbit error entries such as "Failed to get record: decoder: failed to decode payload: EOF" that used up the Logging API write quota for some customers
    Google Cloud2023
  11. IL-0437 Old pods from an unfinished Argo Rollouts canary could not start with a Helm-updated ConfigMap and died in a cascade
    Gunosy2023
  12. IL-0163 An API change that slipped through a gap in automated testing made LKE clusters resize to the wrong number of nodes and added billable worker nodes to customer accounts
    Linode2023
  13. IL-0164 An incremental Linode Kubernetes Engine production update that was missing changes from the previous release made cluster upgrades and node additions fail silently
    Linode2022
  14. IL-0591 Gitpod workspaces in all regions became IO bound and unresponsive for days after the ws39 workspace cluster release
    Gitpod2022
  15. IL-0086 A deployment meant for a CircleCI test Kubernetes cluster also removed Traefik and Calico resources from the production cluster, taking down the CircleCI UI and runner.circleci.com
    CircleCI2021
  16. IL-0088 A validating admission webhook added by a new ingress controller module version made the live-1 cluster reject every new Ingress with an x509 error
    UK Ministry of Justice2020
  17. IL-0034 A crashing stats agent sidecar marked every pod unready and the service mesh pulled all services out of traffic
    Airbnb2019
  18. IL-0149 Upgrades of AKS clusters on Virtual Machine Scale Sets failed with a 'platform image is not available' error after a service update left a backend service unhealthy
    Microsoft Azure2019
  19. IL-0467 Boskos crash-looped after a node drain started it from a buggy StatefulSet image, failing Kubernetes PR jobs for 9 hours
    Kubernetes project2019
  20. IL-0466 Prow scheduled test job pods onto its service cluster, leaving hundreds Pending and starving Prow's own services
    Kubernetes project2019