Cases13 cases

1 to 5 of 13

1. IL-0102 CoreDNS pods on a production EKS cluster ran out of memory under a burst of new pods, endpoints and nodes, taking down the Buildkite web UI, APIs and job dispatch

   Buildkite2026

   Symptoms

   From 22:47 UTC on 25 August 2026 the Buildkite web interface, REST API, Agent API, job queue and notifications stopped working for every customer. No one could create a build from the UI or the REST API, job dispatch was delayed, and some builds that were already running when the outage began stayed stuck until the services recovered.

   Service outageDNS resolution failuresOOMKilledAutoscaler misbehaviour

   Trigger

   At 22:44 UTC an application deploy created a surge of new pods, which used up the free capacity on the existing nodes and made the EKS cluster scale out. Background worker pools with unusually high maximum replica counts were scaling out at the same time and kept asking for more pods while new nodes were still being provisioned. Together they produced an unusually high rate of change in applications, network endpoints and nodes, and with it a sudden rise in demand on CoreDNS.

   Root cause

   Capacity exhaustion

   Component

   CoreDNSContainer memory limitHorizontal pod autoscalerNode pool

   Contributing factor

   No headroomAutomation amplificationRetry stormMonitoring blind spot

   Detection

   The first status post, at 22:55 UTC, eight minutes after the outage began, said something had gone wrong, and at 23:05 UTC the status page said a sitewide outage was being investigated. CoreDNS query duration had been rising as the cluster grew, but a defect in a key monitoring query and too little monitoring of query volume meant this went unnoticed before the outage.

   Remediation

   • paused further application deploys
   • scaled CoreDNS from three to twelve pods
   • raised each CoreDNS pod's memory limit from 512 MiB to 2 GiB
   • expanded the system node pool that runs CoreDNS

   Action items

   • Increase immediate CoreDNS headroom: twelve pods instead of three, a 2 GiB memory limit instead of 512 MiB, and a larger system node pool
   • Enable EKS-managed CoreDNS autoscaling with a tested minimum replica count and enough capacity to spread the replicas across nodes and availability zones
   • Add direct alerts on CoreDNS query duration, query volume, goroutine growth, memory pressure, OOM restarts and available replicas, and track longer-term trends in capacity planning
   • Find other cluster-critical services that lack tested capacity, direct alerting, spread across failure domains, and either safe autoscaling or documented static headroom, and fix them

   Lessons

   A CoreDNS deployment sized once and left at a fixed size can carry a growing cluster for months and then fail on the first burst of churn it cannot absorb. When query processing slows, pending queries take up memory, the pods hit their limits and restart, and client retries keep the service down after the restart. Server-side DNS query duration and volume need their own alerts: here client-side latency was masked by unrelated application speed-ups, and a broken monitoring query hid the server-side trend. Maximum replica counts raised as a temporary measure remove the cap that normally stops runaway scaling when new nodes are slow to arrive.

   View case
2. IL-0155 Some DOKS clusters in NYC1 saw intermittent DNS timeouts and NodeNotReady events because CoreDNS ran on shared-CPU Droplet worker nodes

   DigitalOcean2026

   Symptoms

   From 20:31 UTC on July 9, 2026, DigitalOcean reported a problem with Kubernetes deployments in its NYC1 region. Application workloads could see intermittent DNS failures and NodeNotReady events.

   DNS resolution failuresIntermittent timeoutsNode NotReady

   Root cause

   Capacity exhaustion

   Component

   CoreDNSNode pool

   Contributing factor

   Unknown limit

   Lessons

   Cluster DNS is sensitive to latency, and CoreDNS on shared-CPU nodes can time out intermittently. DigitalOcean's advice is to place CoreDNS on dedicated-CPU node pools and to run enough CoreDNS replicas.

   View case
3. IL-0548 Buildkite's EKS Prometheus server was killed for memory again and again, so services stopped scaling and jobs lagged

   Buildkite2026

   Symptoms

   On 20 May 2026 a subset of Buildkite customers saw delays in notification delivery, build dispatch and the metrics services. The status page first reported delays to notifications for all customers and later narrowed it to a subset of customers.

   High latencyOOMKilledMonitoring outageAutoscaler misbehaviour

   Trigger

   Autoscaling during Buildkite's migration of production services from AWS Fargate to AWS EKS raised the number of pods that Prometheus tracked, and with it the memory the Prometheus server needed.

   Root cause

   Capacity exhaustion

   Component

   PrometheusContainer memory limitNode pool

   Contributing factor

   No resource requests or limits

   Remediation

   • Moved the load of the affected services back to AWS Fargate by following prepared documentation
   • Scaled up the services of a few customers with very large backlogs further by hand

   Action items

   • Upsize the underlying system nodes
   • Set higher requests and limits for the Prometheus server so it can handle more load
   • Review all new EKS resources and set any missing requests and limits
   • Add observability and monitors for EKS pod and node health

   View case
4. IL-0100 PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queries

   PostHog2025

   Symptoms

   On 21 October 2025 at 21:45 UTC a change to timeout handling was deployed to the feature flags service, and two minutes later automated monitoring reported higher error rates. The change was rolled back by 21:49, but error rates stayed high: about 38% of flag evaluation requests in the US datacenter returned errors. At 22:30 Redis metrics showed memory exhaustion on ElastiCache, at 22:35 Postgres connections spiked, and at 22:45 the team found a massive data transfer from Postgres into Redis. The main PostHog application was affected as well, because it shares the Redis instance with the flags service.

   Sustained high node CPUHigh error rateHigh latencyContainer crash loopDatabase overload

   Trigger

   On 21 October a deployment meant to reduce timeout errors (PR #39821); on 24 October the deployment of IP-based rate limiting (PR #40074) that saw all traffic as coming from the load balancer's IP; on 28 October an unrelated routine deployment that rolled the feature flag pods in the US region.

   Root cause

   Capacity exhaustion

   Component

   Node poolPod scheduling and placementIn-memory cacheDatabase

   Contributing factor

   No alert on limitRetry stormIneffective rollbackSingle point of failureNo staged rolloutUnrepresentative staging

   Detection

   On 21 October automated monitoring detected higher error rates two minutes after the deploy. On 24 October no alert existed for 429 errors, and the team learned of the incident from customer reports that began at 18:10 UTC, a 62-minute detection delay. On 28 October on-call received a high error count alert. On 29 October the incident was reported with higher error rates and latency. No alert existed for CPU pressure, so the CPU saturation behind all three incidents was not seen directly.

   Remediation

   • rolled back the timeout handling change within 2 minutes on 21 October, which did not stop the errors
   • throttled connections and Redis writes on 21 October
   • halted the rollout on 28 October, scaled the US flags pods to zero until Redis recovered, then scaled them back gradually
   • rolled back hardware changes on 30 October at 00:03 UTC, which removed most errors but not the latency
   • increased the pod resource requests and adjusted the Kubernetes fleet size so fewer pods ran on each node
   • added retry limits to Envoy between the load balancer and the service
   • moved database connection timeouts and other critical settings to runtime configuration and raised the timeout values
   • added per-node and per-pod CPU alerts at 80% sustained for 5 minutes that page on-call

   Action items

   • Form a strike team from the flags, ingestion and infrastructure teams to review the application and infrastructure for remaining bottlenecks
   • Investigate separating the flags Redis instance from the application Redis instance
   • Before re-enabling ArgoCD sync, align the flags service chart configuration with the values in ArgoCD and define the deployment and Redis strategy
   • Make the feature flags team hard code-owners for flags-related code
   • Build a dashboard of key flag metrics with runbook links and update the feature flag runbooks
   • Add missing alerts on existing service and infrastructure metrics
   • Make readiness checks validate dependencies that degrade under load, for example by pinging the database
   • Rate-limit cache population so it cannot overwhelm Redis
   • Monitor connection pools with automatic throttling near exhaustion and limit concurrent connections
   • Build a production-scale load testing framework
   • Build progressive rollout infrastructure to limit blast radius
   • Document Postgres-to-Redis data flows and write runbooks for data transfer storms

   Lessons

   Nodes that are small relative to pod requests let the scheduler pack many pods on each node, and the resulting CPU saturation showed up three different ways: excessive parallelism, connection pool exhaustion and slow queries. Treating each symptom separately let the problem return; right-sizing the pods and nodes fixed all three. CPU must be alerted on from the start. Unbounded Envoy retries and unbounded cache population amplified the overload, and a Redis instance shared with the main application spread a flags problem to the main application. A rollback did not end the incident, because the data transfer and resource exhaustion it had started kept running.

   View case
5. IL-0082 Losing one CoreDNS pod during a planned EKS node reroll overloaded the remaining CoreDNS pods, causing DNS timeouts and errors across multiple services at RebelMouse

   RebelMouse2025

   Symptoms

   At 13:42 UTC on 19 May 2025, RebelMouse's monitoring tools reported a higher error rate and degraded performance across services. A routine reroll of the EKS nodes had started 27 minutes earlier, at 13:15 UTC.

   DNS resolution failuresHigh latencyHigh error rateService outage

   Trigger

   At 13:15 UTC RebelMouse started its regular reroll of EKS nodes to apply security patches. The planned, automatic termination of an EC2 instance terminated one of the CoreDNS pods.

   Root cause

   Capacity exhaustion

   Component

   CoreDNSNode pool

   Contributing factor

   Change during peak hours

   Detection

   Monitoring tools detected an increased error rate and degraded service performance at 13:42 UTC, and the tech team started investigating at 13:44 UTC.

   Remediation

   • rescheduled the affected CoreDNS pod onto a new EC2 instance
   • scaled up the CoreDNS deployment to more replicas so DNS load was spread evenly across all pods

   Action items

   • Add scale-in protection so CoreDNS pods are not terminated unexpectedly during peak load
   • Review the CoreDNS configuration so that vertical scaling of the pods works
   • Hold regular capacity reviews so the CoreDNS deployment is provisioned for peak load

   View case

By root cause1