Incident Library
Applied filters Root causeConfiguration changeSymptomPod evictionDomainKubernetesClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases5 cases

1 to 5 of 5

  1. IL-0114 An unpinned AWS VPC CNI plugin was updated automatically during a capacity expansion, leaving new pods Pending without IP addresses and stopping CoW Protocol settlements for about 3.5 hours
    CoW Protocol2024
    Symptoms

    At 16:40 UTC on 19 April 2024 CoW Protocol's mainnet production environment stopped producing settlements; the outage officially began then.

    Pods PendingIP address exhaustionPod evictionService outage
    Trigger

    During a cluster capacity expansion, the Infrastructure as Code automation updated the AWS VPC CNI plugin, because the CNI version was not hardcoded in the IaC and the IaC library version had recently changed.

    Root cause
    Configuration change
    Component
    AWS VPC CNINode pool
    Contributing factor
    Unpinned versionOverlapping failures
    Detection

    The CNI version mismatch was identified as the cause at 17:03 UTC, 23 minutes after settlements stopped.

    Remediation
    • upgraded the Kubernetes cluster
    • set up new node groups with a new version of the networking plugin
    Lessons

    Versions in IaC scripts should be specified explicitly so an automation run cannot update a component such as the CNI plugin unintentionally. Changes in production need change management that is easy to revert.

    View case
  2. IL-0019 An EKS Terraform module upgrade shrank node root disks from 100 GB to 20 GB and image pulls failed with no space left on device
    UK Ministry of Justice2023
    Symptoms

    At 13:42 the RootVolUtilisation-Critical alert fired in the high-priority alerts channel. At 14:03 a user reported image pull errors with the message "no space left on device".

    Node disk fullImage pull failuresPod evictionPods stuck in ContainerCreatingIP address exhaustion
    Trigger

    Upgrading the EKS Terraform module to version 18, which changed the node launch template, and draining workloads onto the new nodes it created.

    Root cause
    Configuration change
    Component
    Node root diskNode launch templateNode poolAWS VPC CNICoreDNS
    Contributing factor
    Missed in reviewNo configuration testsNo alert on limit
    Detection

    The RootVolUtilisation-Critical alert fired in the high-priority alerts channel at 13:42, and a user reported "no space left on device" image pull errors at 14:03.

    Remediation
    • Uncordoned the old default node group and drained the new nodes to move workloads back
    • Identified the launch template change that cut the root volume to 20 GB and applied a fix on the manager and live clusters
    • Drained the old node group onto the fixed nodes
    Action items
    • Update the runbook to compare launch template changes during an EKS module upgrade
    • Create a test setup that pulls images of different sizes, like live
    • Update the RootVolUtilisation alert runbook to check the disk space configuration
    • Scale CoreDNS dynamically based on the number of nodes
    • Investigate whether IPv6 can solve the IP prefix starvation problem
    • Add drift testing to identify when a terraform plan shows a change to the launch template
    • Set up logging for the CNI and ipamd logs and alerts on errors related to IP prefix starvation
    Lessons

    The write-up lists review actions rather than lessons. They aim at two gaps: the launch template changes brought by a module upgrade were not compared or drift-tested before rollout, and there were no CNI or ipamd logs or alerts to show the IP prefix starvation the team suspected during the drain.

    View case
  3. IL-0231 Deleting EKS managed node groups also deleted Karpenter's instance profile, and every Stytch worker node shut down
    Stytch2023
    Symptoms

    On 23 February 2023 Stytch's EKS worker nodes went Ready:Unknown one after another, Karpenter failed to launch replacement nodes, and with fewer and fewer pods scheduled the Live Stytch API went down for 14 minutes, returning 5XX on every call.

    Node NotReadyScale-up failurePod evictionPods PendingService outage
    Trigger

    At 14:21 UTC Stytch landed a change to remove unused EKS managed node groups, because it had decided to run its data plane applications fully on nodes provisioned by Karpenter. Because Crossplane resources use an orphan deletion policy, engineers deleted the node groups in the AWS console, first in staging and then in production.

    Root cause
    Configuration change
    Component
    KarpenterNode poolConfiguration templating
    Contributing factor
    Unverified assumptionShared credentialChange with a delayed effectIgnored early signal
    Detection

    Low-urgency alerts that Karpenter could not find the instance profile fired in staging at 16:06 UTC and in production at 17:48 UTC but were not treated as an incident. The incident process started at 20:26 UTC when node launch failures spiked the Karpenter error rate, and Kubernetes monitoring alerts in production followed at 20:44 UTC. The support team flagged the outage at 21:14 UTC.

    Remediation
    • Reverted the node group deletion, which recreated the managed node groups
    • Manually scaled up one of the Live API node groups with more and much larger nodes and removed its taints
    • Removed node selectors from the applications one after another so their pods could be scheduled
    Action items
    • Treat Karpenter errors as high-urgency alerts that page the on-call engineer
    • Tune cluster alerting thresholds to send a high-urgency page earlier
    • Add alerts for nodes that stay NotReady or Unknown for long periods
    • Raise the thresholds of other deployment alerts, such as unschedulable pods and replicas not available
    • Give Karpenter its own IAM instance profile, separate from the one used by the node groups
    • Separate other AWS resources that were shared between uses, specifically IAM and security groups
    • Rework the EKS and Karpenter configuration so system instance profiles are unique, static per cluster and generated by Karpenter
    • Introduce a cloud visualization tool for engineers making infrastructure changes
    • Add live system and product metrics to the status page
    • Ask AWS to confirm the undocumented actions and side effects of deleting a node group
    • Correct the shared use of IAM roles and security groups across the infrastructure
    Lessons

    The IAM role for Karpenter's self-managed nodes should never have been the same as the one for the managed node groups; sharing it is what let the node group deletion break Karpenter, and the roles had been left combined for speed and simplicity when Karpenter was first set up. The hours-long delay between the configuration change and its effect made the incident hard to assess, and AWS confirmed that this delay is expected, so Stytch will allow for it in future IAM instance profile changes.

    View case
  4. IL-0442 Older pods on LIFULL's KEEL Kubernetes cluster were preempted all at once and the external error rate spiked
    LIFULL2021
    Symptoms

    Shortly after the KEEL team applied new PriorityClasses to KEEL, LIFULL's Kubernetes cluster, and saw pods with the low class being preempted as intended, the team received a flood of alerts from external monitoring: the error rate of requests coming from outside the cluster had risen sharply.

    Pod evictionHigh error rate
    Trigger

    The KEEL team deployed three PriorityClasses to the running cluster: low with value: 100, medium with globalDefault: true and value: 1000, and high with value: 10000.

    Root cause
    Configuration change
    Component
    Pod priority and preemptionIngress controller
    Contributing factor
    Unverified assumption
    Detection

    External monitoring sent the team a large number of alerts about a sharp rise in the error rate of requests from outside the cluster.

    Action items
    • Redefine the PriorityClasses as low (-10), medium (0, globalDefault) and high (10), each with a non-preempting variant
    Lessons

    Adding PriorityClasses to a running cluster needs care because every existing pod has priority 0, and a globalDefault class above 0 makes all pods created before it candidates for preemption. PodDisruptionBudgets do not reliably protect pods from preemption. In a multi-tenant cluster, the permission to deploy PriorityClasses should be restricted, and the PriorityClasses tenants may use should be controlled with ResourceQuota or OPA Gatekeeper.

    View case
  5. IL-0004 A new default pod priority let a new Cortex cluster preempt every production ingester in a cascade, taking Grafana Cloud Hosted Prometheus down
    Grafana Labs2019
    Symptoms

    On Friday 19 July 2019 the Grafana Cloud Hosted Prometheus service, which is built on the Cortex project, was down for 26 minutes. New error-budget-based Prometheus alerts paged the on-call engineer within 4 minutes of the start.

    Pod evictionHigh error rateService outageOOMKilled
    Trigger

    An engineer created a new dedicated Cortex cluster for a large customer. Its configuration did not include the new pod priorities, so all its pods got the default medium priority, and the Kubernetes cluster did not have enough free resources to fit it.

    Root cause
    Configuration change
    Component
    Pod priority and preemption
    Contributing factor
    Partial rolloutAutomation amplificationMissed in reviewNo headroomRetry storm
    Detection

    New error-budget-based Prometheus alerts paged the on-call engineer within 4 minutes of the start of the outage.

    Remediation
    • Scaled up the underlying Kubernetes cluster to fit both the new and the production Cortex clusters
    • Diagnosed and fixed the out-of-memory errors in the authenticating reverse proxies in front of Cortex
    Action items
    • Give the production ingesters the high priority class, which should have been done before medium became the default
    • Have more people review changes to configuration that is global to the cluster
    • Require a design document for every configuration change outside a namespace
    • Automate the sizing of the frontend authenticating reverse proxy so it does not run out of memory under overload
    • Investigate the default backoff and scaling behaviour of Prometheus to prevent a stampeding herd of retries
    Lessons

    Medium should not have become the default priority until the production ingesters had the high priority, and that designation should have reached production sooner. The change was considered too minor for a design document and was only discussed in a GitHub issue; cluster-global configuration needs more scrutiny. On the positive side, once Cortex had enough capacity it recovered by itself, and Grafana Loki helped confirm that the ingesters exited gracefully.

    View case
Check your cluster for this

These 5 cases involve Node pool and Pod priority and preemption. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Configuration change5 cases

Component
Node pool3Pod priority and preemption2AWS VPC CNI2Configuration templating1CoreDNS1Ingress controller1Karpenter1Node launch template1Node root disk1
Contributing factor
Unverified assumption2Missed in review2Automation amplification1Change with a delayed effect1Ignored early signal1No alert on limit1No configuration tests1No headroom1Overlapping failures1Partial rollout1Retry storm1Shared credential1Unpinned version1
Organisation
CoW Protocol1Grafana Labs1LIFULL1Stytch1UK Ministry of Justice1
  1. IL-0114 An unpinned AWS VPC CNI plugin was updated automatically during a capacity expansion, leaving new pods Pending without IP addresses and stopping CoW Protocol settlements for about 3.5 hours
    CoW Protocol2024
  2. IL-0019 An EKS Terraform module upgrade shrank node root disks from 100 GB to 20 GB and image pulls failed with no space left on device
    UK Ministry of Justice2023
  3. IL-0231 Deleting EKS managed node groups also deleted Karpenter's instance profile, and every Stytch worker node shut down
    Stytch2023
  4. IL-0442 Older pods on LIFULL's KEEL Kubernetes cluster were preempted all at once and the external error rate spiked
    LIFULL2021
  5. IL-0004 A new default pod priority let a new Cortex cluster preempt every production ingester in a cascade, taking Grafana Cloud Hosted Prometheus down
    Grafana Labs2019