Incident Library
Applied filters ComponentLinux kernelSymptomIntermittent connection failuresFactorMonitoring blind spotClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0112 The daily node rollover put DFDS's EKS nodes on an AMI later recalled for a kernel issue, causing "PLEG is not healthy" NodeNotReady flaps and pod network timeouts
    DFDS2022
    Symptoms

    On 2022-10-31 a planned upgrade of the vpc-cni EKS add-on, started from the deployment pipeline at 13:45 UTC, timed out at 14:10. Only about 10 of the 45 aws-node pods had upgraded, 5 were stuck in Terminating and the other 30 had not been attempted.

    Node NotReadyUpgrade stallIntermittent timeoutsIntermittent connection failures
    Trigger

    At 07:30 UTC on 2022-10-31 the daily node rollover replaced the existing nodes with nodes built from AMI v20221027, the latest AWS EKS node AMI, released on 2022-10-29 and later recalled because of a kernel issue.

    Root cause
    Software defect
    Component
    Node operating system imageLinux kernelkubeletAWS VPC CNINode launch templateTerraform
    Contributing factor
    No staged rolloutAutomation amplificationOverlapping failuresMonitoring blind spotNo health probes
    Detection

    The team noticed nodes flapping between NodeNotReady and NodeReady ("KubeletNotReady PLEG is not healthy") at 14:30 UTC while watching the stalled vpc-cni upgrade, about seven hours after the rollover. Time to detection was 7 hours. Alerting did not catch the NodeNotReady events or the socket growth.

    Remediation
    • cordoned, drained and deleted 15 stuck nodes one at a time to complete the vpc-cni upgrade
    • changed the Terraform infrastructure-modules so a specific AMI could be set
    • rolled the EC2 Auto Scaling group launch configuration back to the previous known good AMI through the pipeline
    Action items
    • Reach out to teams with suggestions about readiness and liveness probes
    • Create a dashboard for readiness and liveness probe coverage
    • Write a wiki article on readiness and liveness probes and their usefulness
    • Modify infrastructure-modules to allow specifying a particular AMI
    • Schedule a discussion on whether to take the latest AMI immediately and how to test new AMIs before they are deployed
    • Implement alerting on abnormal NodeNotReady events and on increased socket use
    • Create a learning path for Kubernetes within the team
    • Collect logs from nodes centrally
    • Create a Prometheus exporter that retrieves the latest available AMI, with a dashboard and alerts
    Lessons

    When a new node AMI has just been rolled out, upgrading other cluster components on the same day makes it hard to tell which change is causing trouble; here the vpc-cni upgrade on the same day confused matters. Taking the latest AMI automatically with no way to pin one meant a rollback first needed a Terraform change. The team also concluded that it needs better communication paths with application teams during incidents.

    View case
  2. IL-0012 CPU limits and an unpatched kernel CFS quota bug throttled multi-threaded services into stalls, failed health checks and timeouts
    Omio2020
    Symptoms

    For about six months, applications on Omio's Google Kubernetes Engine clusters stalled at random: they got stuck, stopped answering health check requests or lost network connections, and nothing the team looked at explained why.

    CPU throttlingProbe failuresHigh error rateHigh latencyIntermittent timeoutsIntermittent connection failures
    Trigger

    Multi-threaded workloads running with CPU limits on nodes whose kernel did not yet have the fix for a known CFS quota bug that causes unnecessary throttling.

    Root cause
    Software defect
    Component
    Container CPU limitLinux kernelkubelet
    Contributing factor
    Unknown limitMonitoring blind spotUnpatched known bug
    Detection

    Months of random stalls and failed health checks led the team to read the cgroup cpu.stat throttling counters inside the pods, which showed far more throttling than the limits should allow.

    Remediation
    • removed CPU limits until the kernel with the CFS bug fix was deployed across all clusters
    Action items
    • Deploy the kernel with the CFS quota bug fix across all clusters
    • Monitor utilisation, saturation and error metrics, API latency and error rates to confirm the change behaves as expected
    Lessons

    A CPU limit is a time quota per 100 ms period, not a share of cores, so a multi-threaded container can be stopped for most of each period long before it uses the cores it was given, and a kernel bug can make that worse. Throttling is invisible in standard Kubernetes metrics and has to be read from the cgroup counters. Removing limits trades away protection against a container that takes a whole node, which is only acceptable when runtimes are tuned.

    View case
Check your cluster for this

These 2 cases involve kubelet and Linux kernel. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Software defect2 cases

Component
kubelet2Linux kernel2Container CPU limit1Node launch template1Node operating system image1Terraform1AWS VPC CNI1
Contributing factor
Monitoring blind spot2Automation amplification1Unpatched known bug1Unknown limit1No health probes1No staged rollout1Overlapping failures1
Organisation
DFDS1Omio1
  1. IL-0112 The daily node rollover put DFDS's EKS nodes on an AMI later recalled for a kernel issue, causing "PLEG is not healthy" NodeNotReady flaps and pod network timeouts
    DFDS2022
  2. IL-0012 CPU limits and an unpatched kernel CFS quota bug throttled multi-threaded services into stalls, failed health checks and timeouts
    Omio2020