Incident Library
Applied filters Root causeSoftware defectFactorAutomation amplificationClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases6 cases

5 to 6 of 6

  1. IL-0112 The daily node rollover put DFDS's EKS nodes on an AMI later recalled for a kernel issue, causing "PLEG is not healthy" NodeNotReady flaps and pod network timeouts
    DFDS2022
    Symptoms

    On 2022-10-31 a planned upgrade of the vpc-cni EKS add-on, started from the deployment pipeline at 13:45 UTC, timed out at 14:10. Only about 10 of the 45 aws-node pods had upgraded, 5 were stuck in Terminating and the other 30 had not been attempted.

    Node NotReadyUpgrade stallIntermittent timeoutsIntermittent connection failures
    Trigger

    At 07:30 UTC on 2022-10-31 the daily node rollover replaced the existing nodes with nodes built from AMI v20221027, the latest AWS EKS node AMI, released on 2022-10-29 and later recalled because of a kernel issue.

    Root cause
    Software defect
    Component
    Node operating system imageLinux kernelkubeletAWS VPC CNINode launch templateTerraform
    Contributing factor
    No staged rolloutAutomation amplificationOverlapping failuresMonitoring blind spotNo health probes
    Detection

    The team noticed nodes flapping between NodeNotReady and NodeReady ("KubeletNotReady PLEG is not healthy") at 14:30 UTC while watching the stalled vpc-cni upgrade, about seven hours after the rollover. Time to detection was 7 hours. Alerting did not catch the NodeNotReady events or the socket growth.

    Remediation
    • cordoned, drained and deleted 15 stuck nodes one at a time to complete the vpc-cni upgrade
    • changed the Terraform infrastructure-modules so a specific AMI could be set
    • rolled the EC2 Auto Scaling group launch configuration back to the previous known good AMI through the pipeline
    Action items
    • Reach out to teams with suggestions about readiness and liveness probes
    • Create a dashboard for readiness and liveness probe coverage
    • Write a wiki article on readiness and liveness probes and their usefulness
    • Modify infrastructure-modules to allow specifying a particular AMI
    • Schedule a discussion on whether to take the latest AMI immediately and how to test new AMIs before they are deployed
    • Implement alerting on abnormal NodeNotReady events and on increased socket use
    • Create a learning path for Kubernetes within the team
    • Collect logs from nodes centrally
    • Create a Prometheus exporter that retrieves the latest available AMI, with a dashboard and alerts
    Lessons

    When a new node AMI has just been rolled out, upgrading other cluster components on the same day makes it hard to tell which change is causing trouble; here the vpc-cni upgrade on the same day confused matters. Taking the latest AMI automatically with no way to pin one meant a rollback first needed a Terraform change. The team also concluded that it needs better communication paths with application teams during incidents.

    View case
  2. IL-0132 A faulty resource cleanup process marked Vertex AI prediction endpoints as deleted worldwide, undeploying customers' models and failing predictions for 4 hours 20 minutes
    Google Cloud2022
    Symptoms

    From 10:10 US/Pacific on 2 June 2022, Vertex AI Online Prediction returned errors for 30% up to 100% of requests per region, depending on how each customer used the service.

    High error rateResources deletedService outage
    Trigger

    A faulty resource cleanup process marked Vertex Prediction Endpoints as deleted, globally.

    Root cause
    Software defect
    Component
    Model request routing
    Contributing factor
    Automation amplificationExcessive blast radius
    Detection

    The first public status update was posted at 13:58 US/Pacific and gave the start as 10:20 US/Pacific; the later mini incident report gives 10:10.

    Remediation
    • restored the Vertex Prediction Endpoints that had been marked as deleted
    View case
Check your cluster for this

These 6 cases involve Node operating system image and Automated capacity manager. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Software defect6 cases

Component
Node operating system image3Automated capacity manager1Cloud load balancer1Application database1Authoritative DNS servers and records1GPU driver1Karpenter1kubelet1Node launch template1Linux kernel1Model request routing1Node pool1Terraform1AWS VPC CNI1
Contributing factor
Automation amplification6Monitoring blind spot2Unverified assumption1Excessive blast radius1Change with a delayed effect1Cache-masked failure1No health probes1No runbook1No safeguard on destructive change1No staged rollout1Overlapping failures1Retry storm1
Organisation
Amazon Web Services1DFDS1Google Cloud1Inflab1Microsoft1OpenAI1
  1. IL-0124 Capacity changes queued while a circuit breaker was engaged ran all at once when it was released, taking multiple OpenAI inference engines out of service and raising API error rates
    OpenAI2026
  2. IL-0150 A latent race condition in DynamoDB's DNS automation left an empty DNS record for dynamodb.us-east-1.amazonaws.com, and the DynamoDB and EC2 launch failures that followed caused EKS, ECS and Fargate container launch failures and cluster scaling delays in us-east-1
    Amazon Web Services2025
  3. IL-0171 Karpenter node auto-repair replaced EKS GPU nodes about 10 minutes after they became Ready, in a loop, because the Bottlerocket NVIDIA AMI lacked libdcgm.so and set AcceleratedHardwareReady to False
    Inflab2025
  4. IL-0147 AKS nodes behind an HTTP proxy failed to provision with CSE exit code 31 on 202508 node images, because a refactoring stopped the proxy from being used when downloading uncached Kubernetes binaries
    Microsoft2025
  5. IL-0112 The daily node rollover put DFDS's EKS nodes on an AMI later recalled for a kernel issue, causing "PLEG is not healthy" NodeNotReady flaps and pod network timeouts
    DFDS2022
  6. IL-0132 A faulty resource cleanup process marked Vertex AI prediction endpoints as deleted worldwide, undeploying customers' models and failing predictions for 4 hours 20 minutes
    Google Cloud2022