Incident Library
Applied filters Root causeConfiguration changeComponentAWS VPC CNISymptomService outageClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0114 An unpinned AWS VPC CNI plugin was updated automatically during a capacity expansion, leaving new pods Pending without IP addresses and stopping CoW Protocol settlements for about 3.5 hours
    CoW Protocol2024
    Symptoms

    At 16:40 UTC on 19 April 2024 CoW Protocol's mainnet production environment stopped producing settlements; the outage officially began then.

    Pods PendingIP address exhaustionPod evictionService outage
    Trigger

    During a cluster capacity expansion, the Infrastructure as Code automation updated the AWS VPC CNI plugin, because the CNI version was not hardcoded in the IaC and the IaC library version had recently changed.

    Root cause
    Configuration change
    Component
    AWS VPC CNINode pool
    Contributing factor
    Unpinned versionOverlapping failures
    Detection

    The CNI version mismatch was identified as the cause at 17:03 UTC, 23 minutes after settlements stopped.

    Remediation
    • upgraded the Kubernetes cluster
    • set up new node groups with a new version of the networking plugin
    Lessons

    Versions in IaC scripts should be specified explicitly so an automation run cannot update a component such as the CNI plugin unintentionally. Changes in production need change management that is easy to revert.

    View case
  2. IL-0105 Replacing the AWS VPC CNI during a Terraform refactoring detached the ALB target groups, and every service in the Kubernetes cluster returned 503
    DFDS2021
    Symptoms

    At 08:29 UTC on 22 June 2021 alerts fired because all services in the Kubernetes cluster were returning 503. No inbound traffic was reaching the cluster. A customer reported the problem at 08:32.

    Service outageHigh error rate
    Trigger

    The pipeline replaced the AWS VPC CNI as part of a refactoring that moved EKS add-ons from kubectl to native Terraform management.

    Root cause
    Configuration change
    Component
    AWS VPC CNICloud load balancerTerraform
    Contributing factor
    Unverified assumptionNo configuration tests
    Detection

    Alerts fired at 08:29 UTC, eight minutes after the VPC CNI was replaced; a customer reported the issue at 08:32.

    Remediation
    • committed a fix that reverted the infrastructure-modules reference
    • re-triggered the pipeline after it failed on an AWS API timeout
    • let the pipeline reattach the target groups to the ALB
    Action items
    • Revert the ALB de-registration delay to the default of 300 seconds
    • Update the roll-over script to set the ALB de-registration delay to zero temporarily before a roll-over
    • Create an ingress test suite
    Lessons

    The change to the de-registration delay had been weighed mainly for performance. Critical pipeline runs should be watched while they run. Everyone should receive the alerts and react to them, and alert messages that were unclear should be reworded.

    View case
Check your cluster for this

These 2 cases involve AWS VPC CNI and Cloud load balancer. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Configuration change2 cases

Component
AWS VPC CNI2Cloud load balancer1Node pool1Terraform1
Contributing factor
Unverified assumption1No configuration tests1Overlapping failures1Unpinned version1
Organisation
CoW Protocol1DFDS1
  1. IL-0114 An unpinned AWS VPC CNI plugin was updated automatically during a capacity expansion, leaving new pods Pending without IP addresses and stopping CoW Protocol settlements for about 3.5 hours
    CoW Protocol2024
  2. IL-0105 Replacing the AWS VPC CNI during a Terraform refactoring detached the ALB target groups, and every service in the Kubernetes cluster returned 503
    DFDS2021