Cases7 cases

1 to 5 of 7

1. IL-0067 Redis filled its persistent volume, corrupted its append-only file and crash-looped on every restart

   BC Government2025sev1

   Symptoms

   The Klamm production application was completely unavailable for about 30 minutes outside working hours. Klamm's queue workers timed out because they could not reach the Redis cache, and the services that depend on it were badly affected.

   Service outageContainer crash loopPersistent volume full

   Trigger

   A large, unexpected spike in I/O from a Klamm Laravel worker sharply increased requests to the Redis cache.

   Root cause

   Capacity exhaustion

   Component

   Persistent volumes and stateful setsIn-memory cache

   Contributing factor

   No capacity planning

   Remediation

   • backed off the Redis StatefulSet and the deployment
   • deleted the corrupted PVC and rebuilt it
   • restarted the StatefulSet and the deployment

   Action items

   • Increase the Klamm Redis cache volume to 100Mi
   • Integrate Sysdig monitoring on the namespace
   • Enable replica pods on the Redis cluster

   View case
2. IL-0006 VPC subnets ran out of IP addresses because the AWS VPC CNI kept IPs on full nodes, and WARM_IP_TARGET changes then blocked pod starts after every pod deletion

   Neon2025

   Symptoms

   On 16 May 2025, in Neon's Kubernetes cluster in AWS us-east-1, new compute pods were scheduled but could not start and failed with IP allocation errors. Each Neon compute is a Postgres database running in a lightweight virtual machine in its own pod, so without new pods, idle customer databases could not be woken up.

   IP address exhaustionPods Pending

   Trigger

   On 16 May the Neon control plane's periodic job that terminates idle computes started failing. Compute creations continued, and the number of active computes rose from about 5k to about 8.1k within minutes. On 19 May, a follow-up change that set WARM_IP_TARGET back to 1 started the second round of failures.

   Root cause

   Capacity exhaustion

   Component

   AWS VPC CNIVPC subnet or pod address rangeCluster autoscaler

   Contributing factor

   Unreviewed defaultsNo capacity planningUnverified assumptionHarmful manual actionIneffective rollbackRetry storm

   Detection

   As cluster-autoscaler added nodes, the team saw IP allocation errors on new pods that were scheduled but could not start. The rate of successful pod starts and VPC subnet metrics were then used to follow the problem.

   Remediation

   • Set WARM_IP_TARGET=1 to release IPs held by nodes without free CPU or memory
   • Fixed the control plane's periodic job so idle computes were terminated again
   • Restarted the aws-node DaemonSet several times, each with a temporary improvement
   • Set WARM_IP_TARGET=0, MINIMUM_IP_TARGET=0 and WARM_ENI_TARGET=0, which let nodes allocate enough IPs and stabilised the cluster
   • On 19 May, set WARM_IP_TARGET back to 0 after setting it to 1 failed again
   • Enlarged the pool of pre-created warm compute pods to absorb failed pod starts

   Action items

   • Open a pull request to AWS CNI to improve how ipamd releases IPs in this situation
   • Follow up with AWS Support about the roughly 100 subnet IPs that were never allocated, for reasons Neon could not explain
   • Change the architecture to support horizontal scaling

   Lessons

   With default settings AWS CNI can keep many IPs on nodes that cannot use them, so the share of subnet IPs assigned to pods does not tell how close the subnet is to running out. WARM_IP_TARGET and WARM_ENI_TARGET follow different logic, a warm IP target of zero switches to the ENI logic, and IPs in cooldown count toward the target, which behaves badly under high pod churn. Changing these settings during an incident without knowing that logic prolonged the outage, and reverting a setting did not bring back the earlier healthy state.

   View case
3. IL-0007 The AWS VPC CNI could not assign IP addresses to pods as an EKS cluster ran out of IPv4 space

   Adevinta2024

   Symptoms

   One evening the platform team that runs SCHIP, Adevinta's internal runtime platform on Kubernetes, was paged because some DaemonSets were failing. Checking the pod status showed this error from the network plugin: name="aws-cni" failed (add): add cmd: failed to assign an IP address to container

   IP address exhaustionPods Pending

   Trigger

   Not stated for the first page. It came in June, during the migration from SCHIP's kube-aws-based clusters to EKS-based clusters; the write-up mentions spikes of pods, such as a job that goes rogue or failing jobs that keep their IPs, as ways a cluster can run out.

   Root cause

   Capacity exhaustion

   Component

   AWS VPC CNIVPC subnet or pod address rangeKarpenter

   Contributing factor

   Unreviewed defaultsNo capacity planning

   Detection

   A pager alert that some DaemonSets were failing. The on-call engineer checked the pod status and found the aws-cni error about failing to assign an IP address.

   Remediation

   • Rolled out VPC CNI custom networking with a secondary CIDR to the smaller clusters from late August
   • Paused the EKS migration in September while available IPs were very low
   • Rolled out custom networking to the rest of the fleet
   • Added the --reserved-enis parameter to the Karpenter deployment

   Action items

   • Use VPC CNI custom networking with a secondary CIDR across the whole fleet
   • Make Karpenter account for custom networking with the --reserved-enis parameter

   Lessons

   For a platform team, the VPC CNI is not ready to scale in production without time spent on capacity planning, and every option has its pros and cons. For large clusters started today, IPv6 is the cheaper and more future-proof option. Do not assume that what works in small clusters will keep working when they grow.

   View case
4. IL-0084 Packet drops on a bare metal provider's vSwitch forced Namespace to rebuild its fra3 region and left global capacity short, and the next day slow NVMe writes on a control plane node drove the Kubernetes API server to very high CPU

   Namespace2024

   Symptoms

   From 17 October 2024, packets through the vSwitch that joined Namespace's bare metal machines at one provider in the fra3 region were dropped at random. Retries in TCP, QUIC and the Namespace control plane hid most of it, but the SLO error budget for latency targets on key interactions began to burn.

   Intermittent connection failuresHigh latencyControl plane lagService outage

   Trigger

   Random packet drops on the bare metal provider's vSwitch from 17 October 2024; for the second outage, a rate of Kubernetes changes that the slow NVMe on one control plane node could no longer keep up with.

   Root cause

   Cloud provider outage

   Component

   Kubernetes API server

   Contributing factor

   No capacity planningNo headroomOverlapping failures

   Detection

   SLO-based monitoring of latency targets showed the error budget burning while retries hid the individual packet drops.

   Remediation

   • diverted all customer traffic from fra3 to other regions
   • connected each provider machine to the zrh1 cluster over Wireguard instead of the vSwitch
   • repaved every fra3 machine with a new identity, certificates and encryption-at-rest keys, dropping regional caches to recover faster
   • cloned existing datasets in a more distributed way to speed up repaving the machines
   • added machines back to the new region in batches and returned the region to the compute pool
   • redirected workloads away from zrh1 during the second outage
   • cloned the failing control plane node's key state to a higher-capacity node that took over its role and IP address

   Action items

   • Credit all instances that ran during the outages back to customers
   • Build up backup regions and solutions so a similar event has options that do not affect customers

   Lessons

   Running hot with a delayed hardware delivery left no room to take a region out, so fixing a provider network fault turned into a global capacity shortage. After one outage the team was anchored on its cause and first looked for network problems in the second outage, which had a different cause: a control plane node whose disk could not keep up with the Kubernetes change rate.

   View case
5. IL-0176 Prometheus in a 235-node Kubernetes cluster crash-looped because over two million series exhausted host memory and the kernel OOM killer chose it first, as Burstable QoS gave it oom_score_adj 999

   LINE2020

   Symptoms

   One Monday morning the author opened the Prometheus dashboard to check a Kubernetes cluster for alerts and could not reach it. The Prometheus pod was in CrashLoopBackOff: kubectl get pods showed it 1/2 ready with 950 restarts over 3d20h on worker3.

   Container crash loopOOMKilledMonitoring outageProbe failures

   Trigger

   After start, the Prometheus time series count rose to 2,162,256 within five minutes and its memory use filled the 8GB host.

   Root cause

   Capacity exhaustion

   Component

   PrometheuskubeletLinux kernel

   Contributing factor

   No resource requests or limitsNo capacity planning

   Detection

   The author found it on a Monday morning when the Prometheus dashboard, opened to check the cluster for alerts, could not be reached.

   Remediation

   • removed most of the unnecessary settings from the Prometheus ConfigMap, after which the container stopped crashing

   Action items

   • Set appropriate resource requests and limits on the Prometheus pod
   • Reduce unneeded metrics by adjusting scrape targets or scrape interval
   • Estimate Prometheus memory needs, provide dedicated hosts with a node label such as deploy-topology: Prometheus, and pin the Prometheus pod to them with a nodeSelector

   Lessons

   Exit code 137 together with an OOM event in the kubelet log means the kernel killed the container, not kubelet. Under node memory pressure the kernel kills the process with the highest score, and kubelet derives each container's oom_score_adj from the pod's QoS class: a container without a memory request in a pod where another container has one ends up Burstable with a score near 1000, so it is killed first. In a large cluster, Prometheus memory needs to be estimated from the expected series count and requested explicitly.

   View case

By root cause2