Incident Library
Applied filters ComponentCluster autoscalerFactorHarmful manual actionDomainKubernetesClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0006 VPC subnets ran out of IP addresses because the AWS VPC CNI kept IPs on full nodes, and WARM_IP_TARGET changes then blocked pod starts after every pod deletion
    Neon2025
    Symptoms

    On 16 May 2025, in Neon's Kubernetes cluster in AWS us-east-1, new compute pods were scheduled but could not start and failed with IP allocation errors. Each Neon compute is a Postgres database running in a lightweight virtual machine in its own pod, so without new pods, idle customer databases could not be woken up.

    IP address exhaustionPods Pending
    Trigger

    On 16 May the Neon control plane's periodic job that terminates idle computes started failing. Compute creations continued, and the number of active computes rose from about 5k to about 8.1k within minutes. On 19 May, a follow-up change that set WARM_IP_TARGET back to 1 started the second round of failures.

    Root cause
    Capacity exhaustion
    Component
    AWS VPC CNIVPC subnet or pod address rangeCluster autoscaler
    Contributing factor
    Unreviewed defaultsNo capacity planningUnverified assumptionHarmful manual actionIneffective rollbackRetry storm
    Detection

    As cluster-autoscaler added nodes, the team saw IP allocation errors on new pods that were scheduled but could not start. The rate of successful pod starts and VPC subnet metrics were then used to follow the problem.

    Remediation
    • Set WARM_IP_TARGET=1 to release IPs held by nodes without free CPU or memory
    • Fixed the control plane's periodic job so idle computes were terminated again
    • Restarted the aws-node DaemonSet several times, each with a temporary improvement
    • Set WARM_IP_TARGET=0, MINIMUM_IP_TARGET=0 and WARM_ENI_TARGET=0, which let nodes allocate enough IPs and stabilised the cluster
    • On 19 May, set WARM_IP_TARGET back to 0 after setting it to 1 failed again
    • Enlarged the pool of pre-created warm compute pods to absorb failed pod starts
    Action items
    • Open a pull request to AWS CNI to improve how ipamd releases IPs in this situation
    • Follow up with AWS Support about the roughly 100 subnet IPs that were never allocated, for reasons Neon could not explain
    • Change the architecture to support horizontal scaling
    Lessons

    With default settings AWS CNI can keep many IPs on nodes that cannot use them, so the share of subnet IPs assigned to pods does not tell how close the subnet is to running out. WARM_IP_TARGET and WARM_ENI_TARGET follow different logic, a warm IP target of zero switches to the ENI logic, and IPs in cooldown count toward the target, which behaves badly under high pod churn. Changing these settings during an incident without knowing that logic prolonged the outage, and reverting a setting did not bring back the earlier healthy state.

    View case
  2. IL-0542 Twingate authentication and authorization requests failed at higher rates after traffic moved onto a smaller GKE cluster
    Twingate2025
    Symptoms

    On 21 March 2025, between 21:10 and 21:41 UTC, part of the authentication and authorization requests to Twingate services failed with elevated error rates.

    High error rate
    Trigger

    Twingate saw anomalies in one of its GKE clusters and scaled down the deployments in that cluster to protect the rest of the system, which moved its traffic to the other clusters.

    Root cause
    Cloud provider outage
    Component
    Cloud load balancerGKE control planeHorizontal pod autoscalerCluster autoscaler
    Contributing factor
    Retry stormNo headroomOverlapping failuresHarmful manual action
    Detection

    Twingate observed anomalies in one of its GKE clusters at the start of the incident.

    Remediation
    • Reintroduced capacity in the cluster that had been scaled down, so traffic was rebalanced across regions
    Action items
    • Keep working with the cloud provider to find the cause of the unexpected timeouts and of the cluster update's impact
    • Run deployments in all regions with the same number of replicas in the HPA configuration
    • Raise the maximum node counts of the cluster node autoscalers to leave more room to scale up
    • Configure the HPA to scale up faster
    View case
Check your cluster for this

These 2 cases involve Cluster autoscaler and Cloud load balancer. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause2

Capacity exhaustion1 case

Component
Cluster autoscalerVPC subnet or pod address rangeAWS VPC CNI
Contributing factor
Unverified assumptionUnreviewed defaultsHarmful manual actionNo capacity planningRetry stormIneffective rollback
Organisation
Neon
  1. IL-0006 VPC subnets ran out of IP addresses because the AWS VPC CNI kept IPs on full nodes, and WARM_IP_TARGET changes then blocked pod starts after every pod deletion
    Neon2025

Cloud provider outage1 case

Component
Cloud load balancerCluster autoscalerGKE control planeHorizontal pod autoscaler
Contributing factor
Harmful manual actionNo headroomOverlapping failuresRetry storm
Organisation
Twingate
  1. IL-0542 Twingate authentication and authorization requests failed at higher rates after traffic moved onto a smaller GKE cluster
    Twingate2025