Cases16 cases

4 to 8 of 16

1. IL-0100 PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queries

   PostHog2025

   Symptoms

   On 21 October 2025 at 21:45 UTC a change to timeout handling was deployed to the feature flags service, and two minutes later automated monitoring reported higher error rates. The change was rolled back by 21:49, but error rates stayed high: about 38% of flag evaluation requests in the US datacenter returned errors. At 22:30 Redis metrics showed memory exhaustion on ElastiCache, at 22:35 Postgres connections spiked, and at 22:45 the team found a massive data transfer from Postgres into Redis. The main PostHog application was affected as well, because it shares the Redis instance with the flags service.

   Sustained high node CPUHigh error rateHigh latencyContainer crash loopDatabase overload

   Trigger

   On 21 October a deployment meant to reduce timeout errors (PR #39821); on 24 October the deployment of IP-based rate limiting (PR #40074) that saw all traffic as coming from the load balancer's IP; on 28 October an unrelated routine deployment that rolled the feature flag pods in the US region.

   Root cause

   Capacity exhaustion

   Component

   Node poolPod scheduling and placementIn-memory cacheApplication database

   Contributing factor

   No alert on limitRetry stormIneffective rollbackSingle point of failureNo staged rolloutUnrepresentative staging

   Detection

   On 21 October automated monitoring detected higher error rates two minutes after the deploy. On 24 October no alert existed for 429 errors, and the team learned of the incident from customer reports that began at 18:10 UTC, a 62-minute detection delay. On 28 October on-call received a high error count alert. On 29 October the incident was reported with higher error rates and latency. No alert existed for CPU pressure, so the CPU saturation behind all three incidents was not seen directly.

   Remediation

   • rolled back the timeout handling change within 2 minutes on 21 October, which did not stop the errors
   • throttled connections and Redis writes on 21 October
   • halted the rollout on 28 October, scaled the US flags pods to zero until Redis recovered, then scaled them back gradually
   • rolled back hardware changes on 30 October at 00:03 UTC, which removed most errors but not the latency
   • increased the pod resource requests and adjusted the Kubernetes fleet size so fewer pods ran on each node
   • added retry limits to Envoy between the load balancer and the service
   • moved database connection timeouts and other critical settings to runtime configuration and raised the timeout values
   • added per-node and per-pod CPU alerts at 80% sustained for 5 minutes that page on-call

   Action items

   • Form a strike team from the flags, ingestion and infrastructure teams to review the application and infrastructure for remaining bottlenecks
   • Investigate separating the flags Redis instance from the application Redis instance
   • Before re-enabling ArgoCD sync, align the flags service chart configuration with the values in ArgoCD and define the deployment and Redis strategy
   • Make the feature flags team hard code-owners for flags-related code
   • Build a dashboard of key flag metrics with runbook links and update the feature flag runbooks
   • Add missing alerts on existing service and infrastructure metrics
   • Make readiness checks validate dependencies that degrade under load, for example by pinging the database
   • Rate-limit cache population so it cannot overwhelm Redis
   • Monitor connection pools with automatic throttling near exhaustion and limit concurrent connections
   • Build a production-scale load testing framework
   • Build progressive rollout infrastructure to limit blast radius
   • Document Postgres-to-Redis data flows and write runbooks for data transfer storms

   Lessons

   Nodes that are small relative to pod requests let the scheduler pack many pods on each node, and the resulting CPU saturation showed up three different ways: excessive parallelism, connection pool exhaustion and slow queries. Treating each symptom separately let the problem return; right-sizing the pods and nodes fixed all three. CPU must be alerted on from the start. Unbounded Envoy retries and unbounded cache population amplified the overload, and a Redis instance shared with the main application spread a flags problem to the main application. A rollback did not end the incident, because the data transfer and resource exhaustion it had started kept running.

   View case
2. IL-0150 A latent race condition in DynamoDB's DNS automation left an empty DNS record for dynamodb.us-east-1.amazonaws.com, and the DynamoDB and EC2 launch failures that followed caused EKS, ECS and Fargate container launch failures and cluster scaling delays in us-east-1

   Amazon Web Services2025

   Symptoms

   At 11:48 PM PDT on 19 October 2025, every system that reached DynamoDB in us-east-1 through the public regional endpoint started getting DNS failures and could not connect. This included customer traffic and internal AWS services that depend on DynamoDB. DynamoDB global table replicas in other regions still answered but fell behind in replication to and from us-east-1.

   DNS resolution failuresHigh error rateScale-up failureIntermittent connection failuresHigh latency

   Trigger

   One DNS Enactor, slowed by unusually high retry delays, applied a much older DNS plan to the regional DynamoDB endpoint just as another Enactor that had applied the newest plan ran its clean-up, which deleted that older plan and with it every IP address of the endpoint.

   Root cause

   Software defect

   Component

   Authoritative DNS servers and recordsApplication databaseCloud load balancer

   Contributing factor

   Automation amplificationRetry stormNo runbook

   Detection

   DNS failures to DynamoDB began immediately at 11:48 PM PDT and the engineering teams of affected AWS services were engaged at once; by 12:38 AM they had identified DynamoDB's DNS state as the source. The later NLB health check problem was detected by monitoring at 6:52 AM.

   Remediation

   • applied temporary mitigations by 1:15 AM that let some internal services reach DynamoDB and repaired key internal tooling
   • restored all DynamoDB DNS information by 2:25 AM through manual operator intervention
   • throttled incoming work and selectively restarted DWFM hosts from 4:14 AM so droplet leases could be re-established
   • reduced the load on Network Manager to bring network configuration propagation times back to normal by 10:36 AM
   • disabled automatic health check failover for NLB at 9:36 AM, and re-enabled it at 2:09 PM after EC2 recovered
   • relaxed the EC2 request throttles from 11:23 AM until all EC2 APIs and launches were normal at 1:50 PM

   Action items

   • Disable the DynamoDB DNS Planner and DNS Enactor automation worldwide
   • Fix the race condition and add protections against applying incorrect DNS plans before re-enabling the DNS automation
   • Add a velocity control to NLB that limits how much capacity a single load balancer can remove when health check failures cause availability zone failover
   • Build an additional EC2 test suite that exercises the DWFM recovery workflow at scale
   • Improve throttling in EC2 data propagation systems so incoming work is rate limited by the size of the waiting queue

   Lessons

   A single DNS record that automation can empty is enough to take a widely used regional endpoint off the network, and every service built on it fails in turn. A check made once at the start of a long operation can be stale by the time the operation writes its result. Recovery of the dependency is not recovery of the dependents: systems that lost state during the outage, such as EC2's droplet leases and network propagation backlog, can collapse under their own catch-up work, and container services that rely on new instance launches keep failing to launch and scale until that work is done.

   View case
3. IL-0006 VPC subnets ran out of IP addresses because the AWS VPC CNI kept IPs on full nodes, and WARM_IP_TARGET changes then blocked pod starts after every pod deletion

   Neon2025

   Symptoms

   On 16 May 2025, in Neon's Kubernetes cluster in AWS us-east-1, new compute pods were scheduled but could not start and failed with IP allocation errors. Each Neon compute is a Postgres database running in a lightweight virtual machine in its own pod, so without new pods, idle customer databases could not be woken up.

   IP address exhaustionPods Pending

   Trigger

   On 16 May the Neon control plane's periodic job that terminates idle computes started failing. Compute creations continued, and the number of active computes rose from about 5k to about 8.1k within minutes. On 19 May, a follow-up change that set WARM_IP_TARGET back to 1 started the second round of failures.

   Root cause

   Capacity exhaustion

   Component

   AWS VPC CNIVPC subnet or pod address rangeCluster autoscaler

   Contributing factor

   Unreviewed defaultsNo capacity planningUnverified assumptionHarmful manual actionIneffective rollbackRetry storm

   Detection

   As cluster-autoscaler added nodes, the team saw IP allocation errors on new pods that were scheduled but could not start. The rate of successful pod starts and VPC subnet metrics were then used to follow the problem.

   Remediation

   • Set WARM_IP_TARGET=1 to release IPs held by nodes without free CPU or memory
   • Fixed the control plane's periodic job so idle computes were terminated again
   • Restarted the aws-node DaemonSet several times, each with a temporary improvement
   • Set WARM_IP_TARGET=0, MINIMUM_IP_TARGET=0 and WARM_ENI_TARGET=0, which let nodes allocate enough IPs and stabilised the cluster
   • On 19 May, set WARM_IP_TARGET back to 0 after setting it to 1 failed again
   • Enlarged the pool of pre-created warm compute pods to absorb failed pod starts

   Action items

   • Open a pull request to AWS CNI to improve how ipamd releases IPs in this situation
   • Follow up with AWS Support about the roughly 100 subnet IPs that were never allocated, for reasons Neon could not explain
   • Change the architecture to support horizontal scaling

   Lessons

   With default settings AWS CNI can keep many IPs on nodes that cannot use them, so the share of subnet IPs assigned to pods does not tell how close the subnet is to running out. WARM_IP_TARGET and WARM_ENI_TARGET follow different logic, a warm IP target of zero switches to the ENI logic, and IPs in cooldown count toward the target, which behaves badly under high pod churn. Changing these settings during an incident without knowing that logic prolonged the outage, and reverting a setting did not bring back the earlier healthy state.

   View case
4. IL-0085 A control plane node of Magic Pages' Kubernetes cluster on Hetzner Cloud ran out of memory, the cluster's own calls then used up the Hetzner API rate limit, and all customer websites were down for about 7.5 hours

   Magic Pages2025

   Symptoms

   At around 09:33 UTC one of the three control plane nodes of the Kubernetes cluster that hosts Magic Pages went offline. Looking at the node showed that it had run out of memory.

   Service outageNode NotReadyCloud API rate limit exhausted

   Trigger

   One of the three control plane nodes ran out of memory and went offline at about 09:33 UTC; the attempts to replace it with hetzner-k3s failed while the cluster's own components were exhausting the Hetzner API rate limit.

   Root cause

   Capacity exhaustion

   Component

   Cloud controller managerPersistent volumes and stateful sets

   Contributing factor

   Unknown limitRetry stormMonitoring blind spotLimited debug accessSingle point of failureNo runbook

   Detection

   The founder found the control plane node offline at about 09:33 UTC. Monitoring alerts about unreachable customer websites started around the time the replacement attempts failed, and about half an hour of investigation identified the exhausted Hetzner API rate limit.

   Remediation

   • powered off all servers of the Kubernetes cluster to stop the API requests so the Hetzner rate limit could recover
   • waited until ratelimit-remaining had built up a buffer before starting recovery
   • restarted the control plane servers
   • started the essential Longhorn components by hand so Longhorn could heal and make storage volumes available
   • restarted the remaining websites by hand

   Action items

   • Monitor the Hetzner API call rate coming from the Kubernetes cluster nodes themselves to get early warning of high usage
   • Get a higher Hetzner API rate limit; Hetzner support was contacted
   • Investigate how to reduce the cluster's reliance on a single API, including alternatives to hetzner-k3s
   • Document and test a procedure for recovering Longhorn after a full cluster shutdown

   Lessons

   The running cluster, not only the deployment tooling, consumed the cloud provider's API quota, and in a failure it consumed it fast enough that the quota never recovered. The default limit of 3600 requests per hour was not enough for failure and recovery scenarios, and API usage from inside the cluster was not monitored. A tool that hides the provider's API errors slows diagnosis. After a full, non-graceful shutdown, Longhorn needed manual intervention to come back, which lengthened the outage.

   View case
5. IL-0024 Expensive cache-backed API calls and uncancelled database requests overloaded MongoDB and took the Hugging Face Hub down three times

   Hugging Face2024

   Symptoms

   At 08:45 CET on 22 April 2024 the Hugging Face Hub became unstable, and by 09:30 it was unresponsive. For most users the website was slow or could not be reached at all.

   Service outageHigh latencyDatabase overload

   Trigger

   Bursts of requests to revision endpoints that read heavily from the *_info_cache collections: a call to https://huggingface.co/api/datasets/lighteval/mmlu/revision/main rose sharply right before the first crash, and a call to https://huggingface.co/api/models/sentence-transformers/all-mpnet-base-v2/revision/main, a model with 300+ associated spaces, lined up with the crash at 21:30 on 23 April.

   Root cause

   Software defect

   Component

   Application databaseModel hubCluster autoscaler

   Contributing factor

   No headroomRetry stormOverlapping failures

   Detection

   Instability started at 08:45 on 22 April and the Hub was unresponsive by 09:30; the team followed MongoDB Atlas metrics and a MongoDB support case. The root cause was found days later from a new graph of requests per URL.

   Remediation

   • Added a firewall rule blocking all requests with a Java/* user agent
   • Disabled the quicksearch feature on the Hub
   • Scaled the MongoDB Atlas cluster by hand from tier 2 to tier 3
   • Switched the Hub to maintenance mode with a 503 page through AWS WAF and blocked incoming traffic
   • Scaled all Hub pods down to 0 and brought them back
   • Set an aggressive rate limit
   • Added timeouts to requests and reworked database queries that were slow or returned large bodies

   Action items

   • Add timeouts to requests based on their usual completion time
   • Optimise requests that took too long or returned a large body, and rework poorly designed database queries
   • Remove old unused indexes from the database
   • Build new dashboards and monitoring metrics
   • Move the *_info_cache collections to Redis
   • Make it possible to abort any database request when the client aborts theirs

   Lessons

   Finding the root cause took two to three days: the existing charts did not show it, and a per-URL request breakdown did. Not cancelling database work when a client goes away turns a strained database into a thundering herd. A close inspection of the database exposed architectural flaws and bottlenecks, such as a collection with large unused indexes, and taught the team which metrics need closer watching to anticipate crashes. Unrelated failures happening at the same time added to the stress and made restoring service harder.

   View case

By root cause4