Cases3 cases

1 to 3 of 3

1. IL-0100 PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queries

   PostHog2025

   Symptoms

   On 21 October 2025 at 21:45 UTC a change to timeout handling was deployed to the feature flags service, and two minutes later automated monitoring reported higher error rates. The change was rolled back by 21:49, but error rates stayed high: about 38% of flag evaluation requests in the US datacenter returned errors. At 22:30 Redis metrics showed memory exhaustion on ElastiCache, at 22:35 Postgres connections spiked, and at 22:45 the team found a massive data transfer from Postgres into Redis. The main PostHog application was affected as well, because it shares the Redis instance with the flags service.

   Sustained high node CPUHigh error rateHigh latencyContainer crash loopDatabase overload

   Trigger

   On 21 October a deployment meant to reduce timeout errors (PR #39821); on 24 October the deployment of IP-based rate limiting (PR #40074) that saw all traffic as coming from the load balancer's IP; on 28 October an unrelated routine deployment that rolled the feature flag pods in the US region.

   Root cause

   Capacity exhaustion

   Component

   Node poolPod scheduling and placementIn-memory cacheDatabase

   Contributing factor

   No alert on limitRetry stormIneffective rollbackSingle point of failureNo staged rolloutUnrepresentative staging

   Detection

   On 21 October automated monitoring detected higher error rates two minutes after the deploy. On 24 October no alert existed for 429 errors, and the team learned of the incident from customer reports that began at 18:10 UTC, a 62-minute detection delay. On 28 October on-call received a high error count alert. On 29 October the incident was reported with higher error rates and latency. No alert existed for CPU pressure, so the CPU saturation behind all three incidents was not seen directly.

   Remediation

   • rolled back the timeout handling change within 2 minutes on 21 October, which did not stop the errors
   • throttled connections and Redis writes on 21 October
   • halted the rollout on 28 October, scaled the US flags pods to zero until Redis recovered, then scaled them back gradually
   • rolled back hardware changes on 30 October at 00:03 UTC, which removed most errors but not the latency
   • increased the pod resource requests and adjusted the Kubernetes fleet size so fewer pods ran on each node
   • added retry limits to Envoy between the load balancer and the service
   • moved database connection timeouts and other critical settings to runtime configuration and raised the timeout values
   • added per-node and per-pod CPU alerts at 80% sustained for 5 minutes that page on-call

   Action items

   • Form a strike team from the flags, ingestion and infrastructure teams to review the application and infrastructure for remaining bottlenecks
   • Investigate separating the flags Redis instance from the application Redis instance
   • Before re-enabling ArgoCD sync, align the flags service chart configuration with the values in ArgoCD and define the deployment and Redis strategy
   • Make the feature flags team hard code-owners for flags-related code
   • Build a dashboard of key flag metrics with runbook links and update the feature flag runbooks
   • Add missing alerts on existing service and infrastructure metrics
   • Make readiness checks validate dependencies that degrade under load, for example by pinging the database
   • Rate-limit cache population so it cannot overwhelm Redis
   • Monitor connection pools with automatic throttling near exhaustion and limit concurrent connections
   • Build a production-scale load testing framework
   • Build progressive rollout infrastructure to limit blast radius
   • Document Postgres-to-Redis data flows and write runbooks for data transfer storms

   Lessons

   Nodes that are small relative to pod requests let the scheduler pack many pods on each node, and the resulting CPU saturation showed up three different ways: excessive parallelism, connection pool exhaustion and slow queries. Treating each symptom separately let the problem return; right-sizing the pods and nodes fixed all three. CPU must be alerted on from the start. Unbounded Envoy retries and unbounded cache population amplified the overload, and a Redis instance shared with the main application spread a flags problem to the main application. A rollback did not end the incident, because the data transfer and resource exhaustion it had started kept running.

   View case
2. IL-0591 Gitpod workspaces in all regions became IO bound and unresponsive for days after the ws39 workspace cluster release

   Gitpod2022

   Symptoms

   From April 11, 2022 at 13:17 UTC customers and Gitpod staff reported that Gitpod workspaces were intermittently becoming unresponsive. CPU usage of nodes and pods looked normal, with nothing maxed out, and at 13:58 UTC the team found that the workspaces were IO bound rather than CPU bound.

   Throughput degradationSustained high node CPUNode NotReadyData loss

   Trigger

   The ws39 generation of workspace clusters, released to all regional clusters on April 8th at 20:08 UTC and to the XL clusters on April 9th at 17:45 UTC, split the 1TB disk into smaller RAID0 disks, changed CPU limiting and workspace networking, and moved to faster machine types.

   Root cause

   Faulty deployment

   Component

   Node root diskContainer CPU limit

   Contributing factor

   No staged rolloutGap in automated testsUnrepresentative stagingUnknown limitOverlapping failuresIneffective rollbackHarmful manual actionLimited debug access

   Detection

   Customers and Gitpod staff reported on April 11th at 13:17 UTC that workspaces were intermittently unresponsive, and the team opened an incident in Slack through its incident.io integration. On April 17th the team noticed increased CPU load on the workspace nodes and opened an internal incident.

   Remediation

   • Shifted workspace starts back to the ws38 clusters on April 11th
   • Built ws39a clusters with c2d machines and local SSDs and moved traffic to them
   • Requested a limit increase from Let's Encrypt
   • Routed EU XL traffic to the regional EU cluster and built ws-eu39bxl to replace the crashed ws-eu39xl
   • Cleaned up workspaces stuck in a stopping state so users could start them again
   • Released ws40 with 1TB boot disks to get more IO bandwidth
   • Removed the workspaces that consumed excessive CPU time for crypto-mining
   • Blocked the fraudulent users behind the crypto-mining from using Gitpod
   • Rolled back the ws-daemon change that broke CPU limiting and shipped the fix in the ws34 clusters
   • Blocked ACME HTTP requests to workspace ports so Let's Encrypt rate limits would not be hit again
   • Rolled out IO limiting with the ws41xl and ws41 clusters and moved traffic to them step by step

   Action items

   • Invest in test automation so that test results stop releases like this one
   • Vary the interval between traffic shifts to new clusters, with more time for larger releases
   • Add a risk assessment stage, to be automated, that governs changeset size and rollout speed of workspace cluster rollouts
   • Reduce the impact teams have on each other
   • Add review criteria that find backwards incompatible changes before pull requests are merged, and split such pull requests
   • Recommend rollback as the primary means of mitigation in the incident procedures, and look at automating it
   • Make the status page show the workspace service as down when it is, and update the playbooks
   • Keep incident communication in a single Slack channel and Google Meet
   • Put 10% of the engineering staff on the on-call rotation from May 9th and use that time on observability, alerting and stability

   Lessons

   Gitpod concluded that it should have committed to rolling back on April 11th instead of building and shipping fixes, which kept customers on degraded workspaces longer than necessary. A large release so soon after the previous one needed a risk assessment and slower traffic shifts, and backwards incompatible changes that block a rollback need to be found before pull requests are merged. Failure-free operations require experience with failure.

   View case
3. IL-0449 Pods on Enigmo's new EKS managed node groups kept restarting or stuck Terminating and the nodes became unusable

   Enigmo2020

   Symptoms

   Two or three days after Enigmo created EKS managed node groups and moved its pods there from the old node groups, the cluster suddenly became unstable: pods restarted again and again or could not stop and stayed around indefinitely, node CPU and memory load rose high, and the nodes became unusable from the cluster.

   Pods stuck in TerminatingSustained high node CPUMemory pressureNode NotReadyUpgrade stall

   Trigger

   Enigmo created EKS managed node groups that used an AMI containing the bug and moved its pods onto them, as preparation for upgrading EKS from 1.15 to 1.16.

   Root cause

   Breaking change in an upgrade

   Component

   Node operating system imageNode pool

   Contributing factor

   Ineffective rollbackChange with a delayed effect

   Remediation

   • Changed the managed node groups' AMI version to the fixed release
   • Ran the managed node group update through CloudFormation
   • Watched the pods across all namespaces during the update
   • Force-deleted every pod that stayed in Terminating with kubectl delete --grace-period=0 --force

   Action items

   • Check the latest releases of the cluster's modules quarterly or monthly and upgrade them regularly

   Lessons

   As long as a team uses Amazon EKS or Kubernetes, it has to keep track of upgrades of the cluster and of modules such as ingress controllers, because a release with breaking changes can be announced without warning after a long stable period. Checking releases and upgrading on a regular schedule keeps the effect of each upgrade on the service small.

   View case

By root cause3