IL-0100 PostHog · 2025-10-21kubernetesaws
Between 21 and 30 October 2025 the PostHog feature flags service had four incidents with more than 14 hours of major impact in total. Three of them had the same root cause: the Kubernetes nodes were too small relative to the pods' resource requests, so Kubernetes packed too many flags pods onto each node and node CPU went above 90%. The fourth, on 24 October, was a separate rate limiting misconfiguration: the tower-governor middleware by default does not trust X-Forwarded-For, so it saw all traffic as coming from the load balancer's IP, and about 97% of requests got 429 errors for 72 minutes until rate limiting was disabled. The rate limiting had been tested only directly against the service, not behind the production load balancers.
Under that CPU pressure, degraded requests made Envoy retry between the load balancer and the service. Each retry created new concurrent requests, each request did several concurrent Redis reads, and on a cache miss the service synchronously loaded the full flag and team state from Postgres and wrote it into Redis. One slow request could therefore fan out into dozens of Redis operations, and the result was bursts of writes that overloaded Redis. At the same time every pod keeps its own Postgres connection pool, and building one needs TLS handshakes, authentication and connection setup, which is expensive on a CPU-bound pod. New pods could not finish this within the 20-second startup timeout and went into crash loops, which reduced healthy capacity further. Because the flags service shares its Redis instance with the main PostHog application, the Redis overload spread to the main app.
On 21 October the triggering deploy was rolled back within 2 minutes, but the data transfer and resource exhaustion continued; the service recovered at 23:28 UTC after connections and Redis writes were throttled (103 minutes). On 28 October the team stopped the rollout, scaled the US pods to zero until Redis recovered from the write storm, and brought them back gradually; full service returned at 21:31 UTC (123 minutes). On 29 to 30 October CPU saturation directly slowed queries for 7 hours 9 minutes. What fixed all three was increasing the pods' resource requests and adjusting the fleet size so fewer, better-resourced pods ran on each node, which brought per-node CPU down. PostHog also added CPU alerts, which had been missing throughout, limited Envoy retries, and configured tower-governor to trust X-Forwarded-For from its load balancer.
On 21 October 2025 at 21:45 UTC a change to timeout handling was deployed to the feature flags service, and two minutes later automated monitoring reported higher error rates. The change was rolled back by 21:49, but error rates stayed high: about 38% of flag evaluation requests in the US datacenter returned errors. At 22:30 Redis metrics showed memory exhaustion on ElastiCache, at 22:35 Postgres connections spiked, and at 22:45 the team found a massive data transfer from Postgres into Redis. The main PostHog application was affected as well, because it shares the Redis instance with the flags service.
On 24 October IP-based rate limiting was deployed to the /flags endpoint at 18:00 UTC, and from 18:01 about 97% of evaluation requests worldwide received 429 (rate limit) errors. No alert existed for 429 errors, so the team learned of it from customer reports, with a 62-minute detection delay; the service recovered at 19:12 when rate limiting was disabled.
On 28 October a routine deployment with no flags code changes rolled the feature flag pods in the US region. From 19:15 the new pods could not connect to Postgres within the 20-second startup timeout and went into crash loops. Redis writes spiked from 19:20, Redis began evicting keys at 19:25 so the cache was effectively gone, the main PostHog app began to suffer at 19:26, and about 34% of US requests failed. On-call was paged by a high error count alert at 19:23.
From 22:30 UTC on 29 October to 05:39 UTC on 30 October queries from the flags service to Redis and Postgres were slow even though the metrics of both showed them healthy. Response times were several times the usual. Node CPU was above 90%, but no CPU alert existed, so this went unnoticed as the cause for more than 7 hours. Rolling back hardware changes at 00:03 removed most of the errors, but the latency stayed.
Customers saw errors and slow responses for all flag types, including read-only evaluations. On 28 October the team first tried the same remediation as on 21 October before finding other ways to reduce parallelism.
Symptom Sustained high node CPU
On 21 October automated monitoring detected higher error rates two minutes after the deploy. On 24 October no alert existed for 429 errors, and the team learned of the incident from customer reports that began at 18:10 UTC, a 62-minute detection delay. On 28 October on-call received a high error count alert. On 29 October the incident was reported with higher error rates and latency. No alert existed for CPU pressure, so the CPU saturation behind all three incidents was not seen directly.
Nodes that are small relative to pod requests let the scheduler pack many pods on each node, and the resulting CPU saturation showed up three different ways: excessive parallelism, connection pool exhaustion and slow queries. Treating each symptom separately let the problem return; right-sizing the pods and nodes fixed all three.
CPU must be alerted on from the start. Unbounded Envoy retries and unbounded cache population amplified the overload, and a Redis instance shared with the main application spread a flags problem to the main application. A rollback did not end the incident, because the data transfer and resource exhaustion it had started kept running.
IL-0202 A burst of queries and database writes exhausted connection pools and memory on Honeycomb's ingest hosts, and Kubernetes crash-loop back-off kept failed hosts offline so ingest capacity stayed too low to recoverHoneycomb · 2023 · sev1same root cause · shared Application database, In-memory cache
IL-0064 CPU-heavy pods packed onto one Google Cloud node drove it to kernel panics, and the crash moved from node to node until the site was downMoonlight · 2019same root cause · shared Application database, Pod scheduling and placement
IL-0102 CoreDNS pods on a production EKS cluster ran out of memory under a burst of new pods, endpoints and nodes, taking down the Buildkite web UI, APIs and job dispatchBuildkite · 2026same root cause · shared Node pool
IL-0155 Some DOKS clusters in NYC1 saw intermittent DNS timeouts and NodeNotReady events because CoreDNS ran on shared-CPU Droplet worker nodesDigitalOcean · 2026same root cause · shared Node pool
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.
same root cause · shared In-memory cache
IL-0082 Losing one CoreDNS pod during a planned EKS node reroll overloaded the remaining CoreDNS pods, causing DNS timeouts and errors across multiple services at RebelMouseRebelMouse · 2025same root cause · shared Node pool
IL-0099 User servers on 2i2c's Earthscope hub took over 15 minutes to start and kernels were killed during a workshop, because a profile option guaranteed 8 GB of memory with no limit on 32 GB nodes2i2c · 2025same root cause · shared Node pool
IL-0027 All CoreDNS pods landed on one node, the node ran out of memory, and DNS for the whole service went downBackpackr · 2025same root cause · shared Node pool
IL-0182 Too few CoreDNS replicas on EKS were overloaded during traffic spikes, raising latency and cutting the network for task pods, and the over-provisioned, tightly spread CoreDNS pods added later blocked Karpenter consolidation이언철 · 2024same root cause · shared Pod scheduling and placement
IL-0110 A Grafana Agent pod's high memory use and crash-looping Hangfire pods drove a Kubernetes node into system OOM and NotReady, twice in one dayDFDS · 2024same root cause · shared Application database
IL-0016 Prometheus on the live cluster ran out of memory and was OOMKilled, and the monitoring node it ran on went NotReadyUK Ministry of Justice · 2023same root cause · shared Node pool
IL-0070 Critical pods on core nodes evicted when Prometheus began scraping about 200 nodes after a dask-gateway scale-up2i2c · 2023same root cause · shared Node pool
IL-0206 Consul agents in GitLab.com's GKE pods were repeatedly OOM-killed, so the Rails pods lost database replica discovery and sent every query to the primaryGitLab · 2022same root cause · shared Application database
IL-0094 Student pods filled the shared nodes of a JupyterHub cluster and starved the hub's core pods, freezing the hub2i2c · 2020same root cause · shared Node pool
IL-0013 The conntrack table filled up on downsized GKE nodes and a high-traffic aggregation proxy saw climbing connection resetsloveholidays · 2020same root cause · shared Node pool
IL-0015 A GKE cluster's /16 pod address range ran out at 256 nodes, leaving new pods Pending and node scale-ups failingloveholidays · 2020same root cause · shared Node pool
IL-0037 Image registry rate-limited a whole region after a crash-looping DaemonSet with imagePullPolicy Always pulled through three NAT instancesDatadog · 2019same root cause · shared Application database
IL-0036 A reindex Job that kept failing exhausted one node after another until the whole cluster was downUniverse · 2017same root cause · shared Pod scheduling and placement