IL-0114 CoW Protocol · 2024-04-19kubernetesawsnetworking
A change to the liquidity fetching logic of CoW Protocol's baseline liquidity caused a performance regression. Higher memory and CPU use got core services evicted frequently, so the team expanded the capacity of the production Kubernetes cluster. The Infrastructure as Code (IaC) run for that expansion also updated the VPC CNI plugin, the AWS component that gives each pod an IP address from the VPC. The CNI version was not hardcoded in the IaC, package versions were not properly pinned, and the IaC library version had recently changed, so the plugin was updated automatically and the result was a CNI version mismatch.
With the mismatched CNI, newly deployed pods never received an IP address and stayed Pending, and updates and pod terminations no longer worked. Settlements stopped at 16:40 UTC. The team identified the CNI version mismatch as the cause at 17:03 UTC, started a cluster upgrade around 19:00 UTC, and set up new node groups with a new version of the networking plugin. Services were restored at 20:08 UTC, about 3.5 hours after the outage began.
At 16:40 UTC on 19 April 2024 CoW Protocol's mainnet production environment stopped producing settlements; the outage officially began then.
Core services had already been evicted often, because a change to the liquidity fetching logic of the baseline liquidity had raised memory and CPU use. After the networking plugin was updated, every restarted pod stayed in Pending because it could not get an IP address, so each restart left the system less available than before. Updates and pod terminations stopped working as well.
Symptom Pods PendingIP address exhaustionPod evictionService outage
Root cause Configuration change
The CNI version mismatch was identified as the cause at 17:03 UTC, 23 minutes after settlements stopped.
–
Versions in IaC scripts should be specified explicitly so an automation run cannot update a component such as the CNI plugin unintentionally. Changes in production need change management that is easy to revert.
IL-0019 An EKS Terraform module upgrade shrank node root disks from 100 GB to 20 GB and image pulls failed with no space left on deviceUK Ministry of Justice · 2023same root cause · shared AWS VPC CNI, Node pool
IL-0017 AWS VPC CNI stopped allocating pod IP addresses on six nodes of the live cluster, leaving new pods in ContainerCreatingUK Ministry of Justice · 2023same root cause · shared AWS VPC CNI, Node pool
IL-0091 Changing the worker node instance type through Terraform recycled all nodes at once and deleted users' production pods togetherUK Ministry of Justice · 2023same root cause · shared Node pool
IL-0109 A retried node rollover on a DFDS EKS cluster set the Auto Scaling group back to 15 instances, and the default termination policy removed every healthy worker node, leaving the cluster with no nodesDFDS · 2023 · sev1same root cause · shared Node pool
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.
same root cause · shared AWS VPC CNI
IL-0055 A failing CronJob with the default concurrency policy piled up thousands of restarting pods and took down a clusterTarget · 2018same root cause · shared Node pool