Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases45 cases

21 to 25 of 45

  1. IL-0141 gke-metadata-server versions 0.4.272 to 0.4.280 flooded Cloud Logging with "Unable to sync sandbox" debug logs on GKE clusters with Workload Identity, using up logging quota and adding billable usage
    Google Cloud2023
    Symptoms

    Google Kubernetes Engine (GKE) customers with Workload Identity enabled saw a high logging rate. Debug log lines containing the string "Unable to sync sandbox" were ingested into Cloud Logging, used up the Cloud Logging ingestion quota and, beyond the free monthly allotment, produced extra billable usage. Workload Identity is opt-in on GKE Standard and always enabled on GKE Autopilot.

    Log floodHigh latency
    Trigger

    GKE clusters ran gke-metadata-server versions 0.4.272 to 0.4.280, which contained an incorrect configuration that wrote "Unable to sync sandbox" debug logs at a high rate.

    Root cause
    Software defect
    Component
    Credential broker for pods
    Contributing factor
    –
    Detection

    The first status update was posted at 13:32 US/Pacific on 2023-11-02, and the updates told customers how to check a cluster from the version annotation on the gke-metadata-server DaemonSet.

    Remediation
    • rolled out a fix across the GKE fleet so the excess logs were no longer ingested into Cloud Logging
    • offered GKE Rapid channel customers a control plane upgrade to 1.28.2-gke.1157000 or above, or 1.27.7-gke.1038000 or above, as a workaround
    View case
  2. IL-0151 A latent defect in the AWS Lambda Frontend, set off when one us-east-1 cell scaled past a size never reached before, raised errors for Lambda and for services that depend on it, including new EKS cluster provisioning
    Amazon Web Services2023
    Symptoms

    From 11:49 AM PDT on 13 June 2023, Lambda function invocations in the Northern Virginia (us-east-1) Region returned more errors and took longer. Customers that triggered functions from asynchronous or streaming event sources saw their event backlogs grow because the events were not being processed.

    High error rateHigh latencyCredentials unavailable
    Trigger

    At 11:49 AM PDT, while adding capacity for normal daily traffic growth, the Lambda Frontend fleet in one cell crossed a capacity threshold that no single cell had reached before.

    Root cause
    Software defect
    Component
    Serverless function platform
    Contributing factor
    Unknown limit
    Detection

    Engineering teams were engaged immediately and began investigating; by 12:26 PM PDT they had identified the latent defect and its effect on provisioning compute capacity.

    Remediation
    • scaled the Lambda Frontend fleet down to a size that no longer triggered the latent defect
    Action items
    • Disable the Lambda Frontend scaling activities that triggered the event
    • Fix the latent bug and deploy the fix to all Regions
    • Complete action items addressing the immediate concern with cellular scaling
    • Bound all cells to a well-tested size to avoid unexpected scaling issues
    Lessons

    A fleet that scales automatically can grow into a size that was never tested, and a defect that only appears at that size stays hidden until ordinary daily growth reaches it. AWS's answer is to cap every cell at a well-tested size. For EKS users, the event shows that provisioning new clusters in us-east-1 depended on Lambda, while clusters that were already running were not affected.

    View case
  3. IL-0174 After an upgrade to Fluent Bit 2.0, logging sidecars that used the Kafka output plugin kept restarting with SIGSEGV because the plugin read a corrupted record as a map without checking its type
    Hyperconnect2023
    Symptoms

    Hyperconnect upgraded the Fluent Bit sidecar of one service with little user impact to Fluent Bit 2.0 and watched it for several days. Only the containers that used the Kafka output plugin were being restarted, each time on SIGSEGV. The stack trace printed at the crash pointed into the Kafka output plugin, but no core file was left behind, because Fluent Bit ran inside Kubernetes.

    Container crash loop
    Trigger

    Upgrading the Fluent Bit logging sidecar to version 2.0 (the 2.0.3 image) on a service with low user impact.

    Root cause
    Software defect
    Component
    Telemetry agent
    Contributing factor
    Limited debug access
    Detection

    The team monitored the upgraded sidecars for several days after the upgrade and saw that the containers using the Kafka output plugin were restarting with SIGSEGV.

    Remediation
    • patched the Fluent Bit Kafka output plugin to skip data that is not a map before calling produce_message()
    • ran a custom build with the patch as the sidecar, which stopped the segmentation faults
    • submitted the patch upstream as fluent-bit pull request 6452, which was merged eight days later
    Lessons

    A segmentation fault in a container can be debugged from its core file, but Kubernetes does not keep one by default: a volume mounted into the pod and a core_pattern pointing into it on the node are needed first. The debug image of the same version, run locally on the same CPU architecture with the matching source checkout, gives a gdb environment that matches production. Core files contain part of the process memory and may hold sensitive data, so they should be shared with care.

    View case
  4. IL-0112 The daily node rollover put DFDS's EKS nodes on an AMI later recalled for a kernel issue, causing "PLEG is not healthy" NodeNotReady flaps and pod network timeouts
    DFDS2022
    Symptoms

    On 2022-10-31 a planned upgrade of the vpc-cni EKS add-on, started from the deployment pipeline at 13:45 UTC, timed out at 14:10. Only about 10 of the 45 aws-node pods had upgraded, 5 were stuck in Terminating and the other 30 had not been attempted.

    Node NotReadyUpgrade stallIntermittent timeoutsIntermittent connection failures
    Trigger

    At 07:30 UTC on 2022-10-31 the daily node rollover replaced the existing nodes with nodes built from AMI v20221027, the latest AWS EKS node AMI, released on 2022-10-29 and later recalled because of a kernel issue.

    Root cause
    Software defect
    Component
    Node operating system imageLinux kernelkubeletAWS VPC CNINode launch templateTerraform
    Contributing factor
    No staged rolloutAutomation amplificationOverlapping failuresMonitoring blind spotNo health probes
    Detection

    The team noticed nodes flapping between NodeNotReady and NodeReady ("KubeletNotReady PLEG is not healthy") at 14:30 UTC while watching the stalled vpc-cni upgrade, about seven hours after the rollover. Time to detection was 7 hours. Alerting did not catch the NodeNotReady events or the socket growth.

    Remediation
    • cordoned, drained and deleted 15 stuck nodes one at a time to complete the vpc-cni upgrade
    • changed the Terraform infrastructure-modules so a specific AMI could be set
    • rolled the EC2 Auto Scaling group launch configuration back to the previous known good AMI through the pipeline
    Action items
    • Reach out to teams with suggestions about readiness and liveness probes
    • Create a dashboard for readiness and liveness probe coverage
    • Write a wiki article on readiness and liveness probes and their usefulness
    • Modify infrastructure-modules to allow specifying a particular AMI
    • Schedule a discussion on whether to take the latest AMI immediately and how to test new AMIs before they are deployed
    • Implement alerting on abnormal NodeNotReady events and on increased socket use
    • Create a learning path for Kubernetes within the team
    • Collect logs from nodes centrally
    • Create a Prometheus exporter that retrieves the latest available AMI, with a dashboard and alerts
    Lessons

    When a new node AMI has just been rolled out, upgrading other cluster components on the same day makes it hard to tell which change is causing trouble; here the vpc-cni upgrade on the same day confused matters. Taking the latest AMI automatically with no way to pin one meant a rollback first needed a Terraform change. The team also concluded that it needs better communication paths with application teams during incidents.

    View case
  5. IL-0139 Pods on GKE clusters with Calico network policy got stuck Terminating or Pending after upgrading to 1.22 or later because of a race condition in the Calico component
    Google Cloud2022
    Symptoms

    After upgrading to GKE 1.22 or later, clusters that use Calico network policy could have pods stuck in Terminating or Pending. Google Cloud posted the incident at 16:01 US/Pacific on 2022-09-15 as pods stuck terminating; from the 2022-09-21 update the description also named pods stuck Pending.

    Pods stuck in TerminatingPods Pending
    Trigger

    Upgrading a GKE cluster that uses Calico network policy to version 1.22 or later.

    Root cause
    Software defect
    Component
    CalicokubeletPod termination
    Contributing factor
    –
    Remediation
    • released a fix in GKE 1.24.4-gke.800 and made it available in 1.23 and 1.22
    • offered an internal patch through Google Cloud Support for affected customers
    • advised customers to restart the kubelet and calico-node to get stuck pods unstuck
    • advised customers without a strong need for Calico to disable Calico network policy
    Action items
    • Upgrade clusters on the Rapid, Regular and Stable release channels running 1.22 or 1.23 to the fixed version automatically
    Lessons

    A race in the Calico component can leave pods stuck Terminating, with FailedKillPod and KillPodSandbox errors that point at the network plugin, not at the workload. Restarting the kubelet and calico-node frees the stuck pods but does not remove the race; the fixed GKE version or Google's internal patch does. Clusters that use the cluster autoscaler are more likely to hit it.

    View case
Check your cluster for this

These 45 cases involve Ingress controller and kubelet. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Software defect45 cases

Component
Ingress controller6kubelet6Node operating system image6Kubernetes API server5Linux kernel5Cloud load balancer4etcd4Node pool4Container CPU limit3Application database3Model request routing3Telemetry agent3Admission webhook2Argo CD2Cluster autoscaler2GPU driver2Container memory limit2Persistent volumes and stateful sets2Pod termination2Terraform2Azure CNI1Calico1Automated capacity manager1cert-manager1Configuration templating1conntrack table1CoreDNS1Credential broker for pods1Client-side DNS resolver and cache1Authoritative DNS servers and records1Elasticsearch or OpenSearch1HAProxy1Horizontal pod autoscaler1Model inference server1Istio1Karpenter1Kubernetes controller manager1kube-proxy1Language runtime settings1Node launch template1Linkerd1Model hub1Pod scheduling and placement1Policy engine1Serverless function platform1TPU1AWS VPC CNI1XLA compiler1
Contributing factor
Unverified assumption8Monitoring blind spot8Automation amplification6Unreviewed defaults6Excessive blast radius5Unpatched known bug5Change with a delayed effect4Ignored early signal4Overlapping failures4Limited debug access3No safeguard on destructive change3No headroom3Single point of failure3Unknown limit2Harmful manual action2No alert on limit2No health probes2No staged rollout2Retry storm2Unrepresentative staging2Credential in configuration1Cache-masked failure1No configuration tests1No resource requests or limits1No runbook1Partial rollout1Ineffective rollback1Deferred upgrades1
Organisation
Google Cloud5Amazon Web Services2DFDS2Hyperconnect2Linode2Microsoft2OpenAI2Airbnb1AirMap1Anthropic1Argo CD users1Barush Mendez1Buffer1Cisco ThousandEyes1Cresta1Cycode1Devsisters1Grafana Labs1Hugging Face1Inflab1Marcin Cuber1Microsoft Azure1Monzo1Nordstrom1Oath1Omio1Palark1PrometheusKube1Ravelin1Skyscanner1thredUP1Toyota Connected1Wiz1Zalando1ZenML1
  1. IL-0058 Argo CD deleted namespaces shared by several applications, leaving about 97 applications stuck Missing and OutOfSync
    Argo CD users2026
  2. IL-0124 Capacity changes queued while a circuit breaker was engaged ran all at once when it was released, taking multiple OpenAI inference engines out of service and raising API error rates
    OpenAI2026
  3. IL-0150 A latent race condition in DynamoDB's DNS automation left an empty DNS record for dynamodb.us-east-1.amazonaws.com, and the DynamoDB and EC2 launch failures that followed caused EKS, ECS and Fargate container launch failures and cluster scaling delays in us-east-1
    Amazon Web Services2025
  4. IL-0104 After a CoreDNS pod was OOMKilled, the talk names three causes: Istio's aggressive DNS cache refresh on one reused client port, kube-proxy not cleaning up conntrack entries of terminated pods, and the 2-minute UDP conntrack timeout
    Cisco ThousandEyes2025
  5. IL-0171 Karpenter node auto-repair replaced EKS GPU nodes about 10 minutes after they became Ready, in a loop, because the Bottlerocket NVIDIA AMI lacked libdcgm.so and set AcceleratedHardwareReady to False
    Inflab2025
  6. IL-0147 AKS nodes behind an HTTP proxy failed to provision with CSE exit code 31 on 202508 node images, because a refactoring stopped the proxy from being used when downloading uncached Kubernetes binaries
    Microsoft2025
  7. IL-0022 Three overlapping infrastructure bugs (context-window misrouting, TPU output corruption and an XLA top-k miscompilation) degraded Claude's responses
    Anthropic2025
  8. IL-0078 Enabling EKS secrets encryption with KMS through Terraform removed worker node IAM permissions and took down all ZenML Pro workspaces
    ZenML2025
  9. IL-0053 Unauthenticated ingress-nginx admission webhook let anyone on the pod network run code and read every Secret in the cluster (IngressNightmare, CVE-2025-1974)
    Wiz2025
  10. IL-0081 API requests timed out during deployment rollouts because the ingress-nginx enable-serial-reloads option held back endpoint updates while a long NGINX reload was running
    Cresta2025
  11. IL-0157 After a release, new nodes could not join Linode Kubernetes Engine clusters with IP ACL enabled, so autoscaling failed on about 20 to 25 clusters
    Linode2025
  12. IL-0138 A failed Nvidia driver build left broken kernel packages on the only Kubernetes master node, which hit a kernel panic on reboot after a power outage and took the homelab cluster down for six days
    Barush Mendez2025
  13. IL-0148 A kube-scheduler regression in Kubernetes v1.27.14, v1.28.10 and v1.29.5 made the scheduler panic on a pod whose node affinity named a nonexistent node, so no pods were scheduled on affected AKS clusters
    Microsoft2024
  14. IL-0131 A new kind of GenerateContent request made Vertex AI prediction API servers segfault, and once enough of them restarted at the same time the serving fleet could not carry the load
    Google Cloud2024
  15. IL-0077 EKS cluster went read-only for over seven hours after Kyverno ephemeralreports filled the etcd database past its size limit
    Marcin Cuber2024
  16. IL-0024 Expensive cache-backed API calls and uncancelled database requests overloaded MongoDB and took the Hugging Face Hub down three times
    Hugging Face2024
  17. IL-0161 A faulty earlier API update made new LKE cluster provisioning in Toronto time out for three days, leaving clusters with no node pools or the wrong number of nodes
    Linode2024
  18. IL-0173 cert-manager's preferredChain logic picked the Let's Encrypt chain through the expired DST Root CA X3, so clients built on an old BoringSSL could not connect to a development server
    Devsisters2024
  19. IL-0065 Any pod could take over the cluster by rewriting Argo CD's unauthenticated Redis cache (CVE-2024-31989)
    Cycode2024
  20. IL-0123 A response buffer allocated anew in a loop made OpenAI's routing layer nodes hit their memory limits and fail readiness checks on a morning with more completions than any day before, causing 502 and 503 errors across all models
    OpenAI2023
  21. IL-0141 gke-metadata-server versions 0.4.272 to 0.4.280 flooded Cloud Logging with "Unable to sync sandbox" debug logs on GKE clusters with Workload Identity, using up logging quota and adding billable usage
    Google Cloud2023
  22. IL-0151 A latent defect in the AWS Lambda Frontend, set off when one us-east-1 cell scaled past a size never reached before, raised errors for Lambda and for services that depend on it, including new EKS cluster provisioning
    Amazon Web Services2023
  23. IL-0174 After an upgrade to Fluent Bit 2.0, logging sidecars that used the Kafka output plugin kept restarting with SIGSEGV because the plugin read a corrupted record as a map without checking its type
    Hyperconnect2023
  24. IL-0112 The daily node rollover put DFDS's EKS nodes on an AMI later recalled for a kernel issue, causing "PLEG is not healthy" NodeNotReady flaps and pod network timeouts
    DFDS2022
  25. IL-0139 Pods on GKE clusters with Calico network policy got stuck Terminating or Pending after upgrading to 1.22 or later because of a race condition in the Calico component
    Google Cloud2022
  26. IL-0175 Lowering a JVM pod's CPU request from 2 to 1 pushed memory past the 90% alert because the JVM treats a cgroup v1 cpu.shares value of exactly 1024 as unset and sized Reactor Netty threads from the node's 36 CPUs
    Hyperconnect2022
  27. IL-0132 A faulty resource cleanup process marked Vertex AI prediction endpoints as deleted worldwide, undeploying customers' models and failing predictions for 4 hours 20 minutes
    Google Cloud2022
  28. IL-0106 A malformed Traefik IngressRoute triggered a Traefik v2 bug once a node rollover restarted every Traefik pod, and all Kubernetes services returned 404
    DFDS2021
  29. IL-0032 Mutating admission controller generated broken JSON patches, silently blocking deploys for a week until a production service ran down to two pods
    Airbnb2020
  30. IL-0146 New AKS clusters with Azure CNI in Central India lost tunnel connectivity because VMs began getting MAC addresses with a 06 prefix, which exposed a dormant bug in bridge-mode Azure CNI
    Microsoft Azure2020
  31. IL-0041 fluent-bit silently stopped shipping application logs from one node to Elasticsearch
    PrometheusKube2020
  32. IL-0029 Slow user-facing services from CPU throttling on containers that used far less CPU than their limit
    Buffer2020
  33. IL-0190 Cortex distributors kept a stuck connection to a dead etcd leader after a Kubernetes upgrade terminated its node, failing writes to Hosted Prometheus for 12 minutes
    Grafana Labs2020
  34. IL-0196 Adding a second Redis replica and lowering memory limits in one RedisFailover change restarted the only Redis pod that held data, and Sentinel promoted the empty new replica, erasing the data
    Palark2020
  35. IL-0012 CPU limits and an unpatched kernel CFS quota bug throttled multi-threaded services into stalls, failed health checks and timeouts
    Omio2020
  36. IL-0143 GKE nodes upgraded to certain 1.13 and 1.14 versions had an elevated number of kernel panics, fixed by releases that downgraded the Container-Optimized OS image
    Google Cloud2019
  37. IL-0062 Intermittent 502 errors behind a GKE Ingress because pods shut down before the load balancer stopped sending them traffic
    Ravelin2019
  38. IL-0011 HAProxy ingress piled requests onto the same pods because a Go template visited the backend map in sorted pod IP order
    Skyscanner2019
  39. IL-0030 Most nodes of a cluster went NotReady together for 15 minutes while the API server sat behind an AWS ELB
    Zalando2019
  40. IL-0050 Pods on Kubernetes failed to resolve an external Redis hostname because of ndots:5, the musl resolver in Alpine images, and Netty's DNS resolver
    Toyota Connected2018
  41. IL-0051 A third-party log shipping client leaked file descriptors with every new container until nodes failed with too many open files
    thredUP2018
  42. IL-0074 All AirMap platform services down as worker nodes went NotReady and pods crash-looped on network and DNS failures
    AirMap2018
  43. IL-0038 kubelet rollout hit a Kubernetes bug and left every node in a cluster NotReady, triggering mass pod eviction
    Oath2018
  44. IL-0072 High etcd write latency made the API servers return stale data, and the production cluster flipped between two views of its state
    Nordstrom2017
  45. IL-0003 linkerd lost Kubernetes service discovery after an etcd reconfiguration, and restarting every linkerd instance turned failed payments into a full platform outage
    Monzo2017