Cases2 cases

1 to 2 of 2

1. IL-0006 VPC subnets ran out of IP addresses because the AWS VPC CNI kept IPs on full nodes, and WARM_IP_TARGET changes then blocked pod starts after every pod deletion

   Neon2025

   Symptoms

   On 16 May 2025, in Neon's Kubernetes cluster in AWS us-east-1, new compute pods were scheduled but could not start and failed with IP allocation errors. Each Neon compute is a Postgres database running in a lightweight virtual machine in its own pod, so without new pods, idle customer databases could not be woken up.

   IP address exhaustionPods Pending

   Trigger

   On 16 May the Neon control plane's periodic job that terminates idle computes started failing. Compute creations continued, and the number of active computes rose from about 5k to about 8.1k within minutes. On 19 May, a follow-up change that set WARM_IP_TARGET back to 1 started the second round of failures.

   Root cause

   Capacity exhaustion

   Component

   AWS VPC CNIVPC subnet or pod address rangeCluster autoscaler

   Contributing factor

   Unreviewed defaultsNo capacity planningUnverified assumptionHarmful manual actionIneffective rollbackRetry storm

   Detection

   As cluster-autoscaler added nodes, the team saw IP allocation errors on new pods that were scheduled but could not start. The rate of successful pod starts and VPC subnet metrics were then used to follow the problem.

   Remediation

   • Set WARM_IP_TARGET=1 to release IPs held by nodes without free CPU or memory
   • Fixed the control plane's periodic job so idle computes were terminated again
   • Restarted the aws-node DaemonSet several times, each with a temporary improvement
   • Set WARM_IP_TARGET=0, MINIMUM_IP_TARGET=0 and WARM_ENI_TARGET=0, which let nodes allocate enough IPs and stabilised the cluster
   • On 19 May, set WARM_IP_TARGET back to 0 after setting it to 1 failed again
   • Enlarged the pool of pre-created warm compute pods to absorb failed pod starts

   Action items

   • Open a pull request to AWS CNI to improve how ipamd releases IPs in this situation
   • Follow up with AWS Support about the roughly 100 subnet IPs that were never allocated, for reasons Neon could not explain
   • Change the architecture to support horizontal scaling

   Lessons

   With default settings AWS CNI can keep many IPs on nodes that cannot use them, so the share of subnet IPs assigned to pods does not tell how close the subnet is to running out. WARM_IP_TARGET and WARM_ENI_TARGET follow different logic, a warm IP target of zero switches to the ENI logic, and IPs in cooldown count toward the target, which behaves badly under high pod churn. Changing these settings during an incident without knowing that logic prolonged the outage, and reverting a setting did not bring back the earlier healthy state.

   View case
2. IL-0193 A GCP persistent disk degradation took down a Cassandra StatefulSet whose OrderedReady policy blocked recovery, and Metrictank could not reconnect to the recreated pods, causing a 23-hour Graphite query outage

   Grafana Labs2019

   Symptoms

   An alert fired because some Metrictank instances could not drain their write queue, which meant they could not write to Cassandra. Google had not yet published an incident, but a team member's Discord server, also hosted on GCP, was down, which pointed at GCP; Google confirmed an SSD persistent disk incident about 90 minutes after it began.

   High error rateNode NotReadyPods PendingService outage

   Trigger

   A Google Cloud incident in which SSD persistent disks performed so poorly that many I/O operations failed for an extended period.

   Root cause

   Cloud provider outage

   Component

   Persistent volumes and stateful setsApplication databaseNode root disk

   Contributing factor

   Overlapping failuresUnpatched known bugUnverified assumption

   Detection

   An alert that Metrictank write instances could not flush their write queue to Cassandra. Error rates alone rose too slowly to notice quickly, because most queries were served from memory.

   Remediation

   • Added compute nodes hoping new nodes could attach the disks, which did not work
   • Deleted and recreated the Cassandra StatefulSet with an updated pod management policy, so all pods but one started
   • Raised Kafka topic retention to avoid data loss during the long downtime
   • Temporarily added nodes so Metrictank restarts were not slowed by CPU starvation
   • Restarted all Metrictank instances, which took more than 20 hours, so they connected to the new Cassandra IPs

   Action items

   • Re-instantiate the gocql connection in Metrictank when it is detected dead, so it reconnects even when all Cassandra IPs change
   • Fix the Metrictank write-queue deadlock by updating an internal data structure with atomics instead of a lock

   Lessons

   An OrderedReady StatefulSet cannot recover past a single pod whose volume will not attach; when every pod is down, the ordering that protects normal start-up blocks the whole cluster. A client that learns peer addresses only from live peers needs a way to go back to the service name when all peers are replaced at once. A long-tolerated bug with a config workaround can surface at the worst time; this one turned out to have a simple fix. Writing incoming data to a Kafka write-ahead log first meant all data could still be ingested after recovery, and Kafka on non-SSD disks escaped the disk incident. With both Metrictank bugs fixed, the team expects a similar event to take less than 3 hours to recover from instead of 23.

   View case

By root cause2