Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases14 cases

11 to 14 of 14

  1. IL-0086 A deployment meant for a CircleCI test Kubernetes cluster also removed Traefik and Calico resources from the production cluster, taking down the CircleCI UI and runner.circleci.com
    CircleCI2021
    Symptoms

    At 15:35 EST on 11 March 2021 the CircleCI web UI and runner.circleci.com became unavailable. The API, which uses a separate ingress point, stayed reachable, and builds kept running.

    Service outageResources deleted
    Trigger

    An update deployed to a test Kubernetes cluster also removed Traefik and Calico resources from CircleCI's primary production cluster.

    Root cause
    Faulty deployment
    Component
    Ingress controllerCalico
    Contributing factor
    Excessive blast radiusNo runbookMonitoring blind spot
    Detection

    CircleCI engineers were alerted and began investigating. The status page posted its first notice at 20:45 UTC.

    Remediation
    • restored Traefik in the production cluster
    • restored Calico in the production cluster, after which pods served customer requests again
    Action items
    • Improve network-level documentation and runbooks to cover every layer of the Kubernetes networks
    • Improve monitoring and alerting for the network infrastructure
    • Add guardrails that isolate the production and staging clusters so they cannot interfere with each other
    Lessons

    A deployment path that can reach both a test cluster and production can remove shared networking components from production. When the ingress controller and the CNI are both affected, restoring the ingress alone is not enough: engineers had to find the link between Calico and Traefik and restore Calico before pods served traffic again. Time went to DNS and AWS red herrings because nothing linked the outage to the test deployment; CircleCI's answer is better network runbooks, network alerting and guardrails between clusters.

    View case
  2. IL-0088 A validating admission webhook added by a new ingress controller module version made the live-1 cluster reject every new Ingress with an x509 error
    UK Ministry of Justice2020
    Symptoms

    At 12:39 on 7 September 2020 the Cloud Platform team found that the Ingress API of the live-1 cluster was refusing every POST request. A user who provisioned a new service could not create an ingress into the cluster. The incident was declared at 12:54.

    Deploys failing
    Trigger

    Version 0.1.0 of the team's ingress controller module started creating a ValidatingWebhookConfiguration resource, and one new 0.1.0 ingress controller in the live-1 cluster failed to be created because of AWS resource limits.

    Root cause
    Faulty deployment
    Component
    Admission webhookIngress controller
    Contributing factor
    Single point of failure
    Detection

    The Ingress API was found refusing 100% of POST requests at 12:39; the incident was declared at 12:54.

    Remediation
    • removed by hand all the additional admission controllers created by module version 0.1.0
    • removed the admission webhook from the ingress controller module and released version 0.1.1
    • upgraded every ingress module on version 0.1.0 to 0.1.1
    Lessons

    Turning on the validating webhook in the shared ingress controller module made the webhook a single point of failure for every ingress controller pod in the ingress-controllers namespace: when one controller could not come up, no team could create an ingress.

    View case
  3. IL-0034 A crashing stats agent sidecar marked every pod unready and the service mesh pulled all services out of traffic
    Airbnb2019
    Symptoms

    An operation to upgrade the stats agent in Kubernetes was announced in advance, with a warning that metrics might be unavailable to services for a few minutes. A few minutes later a reply came in saying that all of the sender's Kubernetes services seemed to be down.

    Probe failuresService outage
    Trigger

    A scheduled upgrade of the stats agent sidecar, during which the agent container crashed.

    Root cause
    Faulty deployment
    Component
    Telemetry agentService discovery
    Contributing factor
    Unverified assumptionExcessive blast radius
    Detection

    A reply to the operation announcement, a few minutes later, said that all of the sender's Kubernetes services appeared to be down.

    Action items
    • Mark logging, tracing and stats agent containers as non-critical and leave them out of the service mesh's pod readiness check
    Lessons

    Every container in a pod affects the pod's readiness, so each sidecar needs a deliberate decision about how its health should count. A sidecar that crashes is not ready even when it has no probes, and any logic that requires all containers to be ready will take the pod out of traffic.

    View case
  4. IL-0149 Upgrades of AKS clusters on Virtual Machine Scale Sets failed with a 'platform image is not available' error after a service update left a backend service unhealthy
    Microsoft Azure2019
    Symptoms

    From 07:20 UTC on 28 September 2019, customers running Azure Kubernetes Service clusters on Virtual Machine Scale Sets (VMSS) got failures when they tried any cluster upgrade. The error read: "The platform image 'microsoft-aks:aks:aks-ubuntu-1604-201909:2019.08.21' is not available. Verify that all fields in the storage profile are correct." The issue title calls the error PlatformImageNotFound.

    Upgrade stall
    Trigger

    A recent service update affected instances of a backend service, which became unhealthy and stopped requests from completing.

    Root cause
    Faulty deployment
    Component
    Node operating system imageNode pool
    Contributing factor
    –
    Detection

    Customers received failure messages when they tried to upgrade their clusters.

    Remediation
    • published az vmss update commands that set the scale set's image version to 2019.09.25 and SKU to aks-ubuntu-1604-201909, after which the upgrade could be run again
    • mitigated affected clusters by hand (AKS and VMSS teams)
    • rolled out hot fixes to all regions, completed by 15 October 2019
    View case
Check your cluster for this

These 14 cases involve Node pool and Ingress controller. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Faulty deployment14 cases

Component
Node pool5Ingress controller3Telemetry agent3Cluster autoscaler2Admission webhook1In-memory cache1Calico1Cloud load balancer1Application database1Descheduler1GPU driver1Managed Kubernetes control plane and provider API1Node operating system image1Pod termination1Service discovery1
Contributing factor
Unverified assumption3Excessive blast radius3Monitoring blind spot3Unreviewed defaults2No runbook2Automation amplification1Change during peak hours1No alert on limit1No safeguard on destructive change1No headroom1Health check tied to a dependency1Retry storm1Missed in review1Single point of failure1Gap in automated tests1
Organisation
Linode5Microsoft Azure2UK Ministry of Justice2Airbnb1CircleCI1Google Cloud1OpenAI1Yonghyun Kim1
  1. IL-0136 A deployment that cleared an internal service's caches at peak traffic sent 20 times the normal load to the Azure OpenAI metadata backend in Sweden Central, and health checks that depended on its database restarted instances in a loop for almost six hours
    Microsoft Azure2026
  2. IL-0166 A compute host update removed the credential parser used to decode StackScripts, so new Linodes and every new LKE and LKE-E node failed to boot and autoscaling and node recycling failed in all regions
    Linode2026
  3. IL-0167 A phased rollout of an LKE software release in Osaka stalled LKE-E cluster creation, version upgrades and control plane ACL changes
    Linode2026
  4. IL-0160 A repaired legacy GPU monitoring script sent a firmware query that triggers a race condition in the GPU System Processor of NVIDIA RTX 4000 Ada GPUs, locking up GPU Linodes and LKE worker nodes in several regions
    Linode2026
  5. IL-0188 Rolling updates on EKS returned 502 errors for about 30 seconds because Kubernetes deleted old pods as soon as new pods passed readiness, before the ALB had marked the new pod targets healthy
    Yonghyun Kim2025
  6. IL-0121 A faulty code change to a control plane cluster meant for low-priority workloads degraded the production inference workloads it also managed
    OpenAI2025
  7. IL-0020 During an EKS 1.30 upgrade the descheduler tried to reschedule every pod on the cordoned nodes at once and production workloads went down
    UK Ministry of Justice2025
  8. IL-0142 GKE clusters on 1.24 and later wrote unexpected fluentbit error entries such as "Failed to get record: decoder: failed to decode payload: EOF" that used up the Logging API write quota for some customers
    Google Cloud2023
  9. IL-0163 An API change that slipped through a gap in automated testing made LKE clusters resize to the wrong number of nodes and added billable worker nodes to customer accounts
    Linode2023
  10. IL-0164 An incremental Linode Kubernetes Engine production update that was missing changes from the previous release made cluster upgrades and node additions fail silently
    Linode2022
  11. IL-0086 A deployment meant for a CircleCI test Kubernetes cluster also removed Traefik and Calico resources from the production cluster, taking down the CircleCI UI and runner.circleci.com
    CircleCI2021
  12. IL-0088 A validating admission webhook added by a new ingress controller module version made the live-1 cluster reject every new Ingress with an x509 error
    UK Ministry of Justice2020
  13. IL-0034 A crashing stats agent sidecar marked every pod unready and the service mesh pulled all services out of traffic
    Airbnb2019
  14. IL-0149 Upgrades of AKS clusters on Virtual Machine Scale Sets failed with a 'platform image is not available' error after a service update left a backend service unhealthy
    Microsoft Azure2019