Cases18 cases

1 to 5 of 18

1. IL-0102 CoreDNS pods on a production EKS cluster ran out of memory under a burst of new pods, endpoints and nodes, taking down the Buildkite web UI, APIs and job dispatch

   Buildkite2026

   Symptoms

   From 22:47 UTC on 25 August 2026 the Buildkite web interface, REST API, Agent API, job queue and notifications stopped working for every customer. No one could create a build from the UI or the REST API, job dispatch was delayed, and some builds that were already running when the outage began stayed stuck until the services recovered.

   Service outageDNS resolution failuresOOMKilledAutoscaler misbehaviour

   Trigger

   At 22:44 UTC an application deploy created a surge of new pods, which used up the free capacity on the existing nodes and made the EKS cluster scale out. Background worker pools with unusually high maximum replica counts were scaling out at the same time and kept asking for more pods while new nodes were still being provisioned. Together they produced an unusually high rate of change in applications, network endpoints and nodes, and with it a sudden rise in demand on CoreDNS.

   Root cause

   Capacity exhaustion

   Component

   CoreDNSContainer memory limitHorizontal pod autoscalerNode pool

   Contributing factor

   No headroomAutomation amplificationRetry stormMonitoring blind spot

   Detection

   The first status post, at 22:55 UTC, eight minutes after the outage began, said something had gone wrong, and at 23:05 UTC the status page said a sitewide outage was being investigated. CoreDNS query duration had been rising as the cluster grew, but a defect in a key monitoring query and too little monitoring of query volume meant this went unnoticed before the outage.

   Remediation

   • paused further application deploys
   • scaled CoreDNS from three to twelve pods
   • raised each CoreDNS pod's memory limit from 512 MiB to 2 GiB
   • expanded the system node pool that runs CoreDNS

   Action items

   • Increase immediate CoreDNS headroom: twelve pods instead of three, a 2 GiB memory limit instead of 512 MiB, and a larger system node pool
   • Enable EKS-managed CoreDNS autoscaling with a tested minimum replica count and enough capacity to spread the replicas across nodes and availability zones
   • Add direct alerts on CoreDNS query duration, query volume, goroutine growth, memory pressure, OOM restarts and available replicas, and track longer-term trends in capacity planning
   • Find other cluster-critical services that lack tested capacity, direct alerting, spread across failure domains, and either safe autoscaling or documented static headroom, and fix them

   Lessons

   A CoreDNS deployment sized once and left at a fixed size can carry a growing cluster for months and then fail on the first burst of churn it cannot absorb. When query processing slows, pending queries take up memory, the pods hit their limits and restart, and client retries keep the service down after the restart. Server-side DNS query duration and volume need their own alerts: here client-side latency was masked by unrelated application speed-ups, and a broken monitoring query hid the server-side trend. Maximum replica counts raised as a temporary measure remove the cap that normally stops runaway scaling when new nodes are slow to arrive.

   View case
2. IL-0246 New pods on Adyen's big data Kubernetes nodes could not start, failing with "Cilium API client timeout exceeded"

   Adyen2026

   Symptoms

   On Adyen's production big data Kubernetes clusters, new pods on some nodes could not start because setting up their network through Cilium timed out. Once one pod had timed out, every other pod trying to start on the same node timed out as well.

   Pods stuck in ContainerCreating

   Trigger

   Heavy data workloads landed on nodes where the interval of Cilium's periodic garbage collection of expired connection tracking entries had grown, during a quiet period, toward its 12-hour maximum.

   Root cause

   Configuration change

   Component

   Cilium

   Contributing factor

   Change with a delayed effectUnreviewed defaultsRetry stormNo alert on limit

   Remediation

   • Set a fixed connection tracking garbage collection interval, conntrackGCInterval: 60s, and rolled it out to the Cilium DaemonSet

   Action items

   • Alert on the connection tracking garbage collection duration and table size metrics

   Lessons

   Scaling parameters can interact with other logic months later as traffic grows: the larger table that solved capacity limits on 512GB nodes interacted with the garbage collection auto-scaling on the 2TB nodes. Logs only showed symptoms, so profiling and tracing across the container runtime, the CNI plugin, the Cilium agent and the kernel were needed, and simple arithmetic with measured numbers pointed at the cause early. Auto-scaling algorithms need bounds, and the 12-hour maximum interval was too long for this workload. A timeout protects the caller but does not cancel the work behind it. Adyen recommends that Cilium deployments with dynamic workloads alert on cilium_datapath_conntrack_gc_duration_seconds and cilium_datapath_conntrack_gc_entries and set conntrackGCInterval: 60s.

   View case
3. IL-0499 Discord voice and video calls hung at Awaiting Endpoint for three hours after 17% of sessions stopped at once

   Discord2026

   Symptoms

   From 12:13 to 15:30 PDT on 25 March 2026, voice and video on Discord were heavily degraded. Users mostly could not start or join calls and saw "Awaiting Endpoint" in their call status.

   Service outage

   Trigger

   A routine change to tune the sessions service cluster on Kubernetes gave each pod more CPU and memory and lowered the pod count in proportion. When it was deployed to the first zone at 12:13 PDT, Kubernetes terminated 50% of that zone's pods, and the termination grace period ran out before the sessions could be handed off.

   Root cause

   Configuration change

   Component

   Pod terminationCoordination and key-value storeDatabase proxy and connection pooler

   Contributing factor

   Retry stormUnknown limit

   Remediation

   • Set low syncer spawn rate limits in the calls, streams and guilds services and restarted the voice syncers cluster
   • Restarted voice syncers instances one at a time, and four of the first five recovered fully
   • Brought 15 new voice syncers instances online, doubling capacity and halving the syncers per instance
   • Restarted the last backed-up voice syncers instance, leaving the doubled cluster fully healthy

   Action items

   • Add a validating admission webhook that rejects a scale-down of Elixir workloads until each pod has handed off its processes
   • Replace the Holster.Pool supervisor with a PartitionSupervisor so outgoing connections are created and checked out concurrently
   • Move gun connection process lifecycle management into Holster.Pool so the gun connection supervisor is no longer used
   • Tune the upstream rate limits that call into voice syncers, with load shedding tooling that can force them
   • Add rate limits on endpoint selection in voice syncers to limit outbound connection creation
   • Monitor the HTTP connection pooling mailboxes
   • Review and expand instrumentation and logging for service discovery announcement and voice syncer RPCs to SFUs
   • Monitor the most critical processes across all services instead of aggregate metrics such as overall mailbox health
   • Bring more workloads into Kubernetes so capacity can be scaled with a single command instead of provisioning VMs by hand
   • Re-architect the A/V system to scale its control plane services horizontally across multiple regions

   Lessons

   Discord says a large enough traffic spike will find a bottleneck somewhere; in earlier large-load incidents its ScyllaDB databases limited recovery, and this time the bottleneck moved to the voice services. Facing a capacity constraint, the team asks whether to increase supply, such as adding services, or reduce demand, such as rate limiting inbound traffic. During an incident it matters to know the toolbox, and Elixir and the BEAM give engineers many introspection options. Any slowdown in the hot loop of a single-threaded process can degrade both the service and the user experience.

   View case
4. IL-0063 VPN outage after recreating a Redis StatefulSet restarted the whole Redis cluster without readiness checks

   AdGuard VPN2025

   Symptoms

   The day began with an alert that several instances of the Redis cluster were running out of disk space. That part was fixed quickly. The outage came an hour later: right after the Redis StatefulSet was deleted and recreated at 8:10 UTC, Kubernetes restarted every Redis pod within about five minutes, and problems started to appear.

   Service outageOOMKilledProbe failuresHigh latency

   Trigger

   To make Git match the volumes that had been resized by hand, the Redis StatefulSet was deleted with orphan (non-cascading) deletion and recreated with the new storage template; Kubernetes reconciled the new spec by restarting the entire Redis cluster.

   Root cause

   Configuration change

   Component

   Persistent volumes and stateful setsIn-memory cacheContainer memory limit

   Contributing factor

   No health probesUnverified assumptionSingle point of failureRetry stormNo headroom

   Detection

   An alert about Redis instances running out of disk space started the work; the outage itself showed up as backend errors about Redis connections right after the StatefulSet was recreated.

   Remediation

   • made the Redis masters forget obsolete node identities, reset the affected replicas and rejoined them, then waited for replication to catch up
   • identified the component that was actually running out of memory, scaled the backend service and raised memory limits for that component
   • temporarily relaxed probes so backend instances took traffic immediately instead of flapping between ready and not ready
   • reverted the temporary workarounds at 10:00 UTC

   Action items

   • Add startup probes so a Redis pod is not marked ready until its dataset is loaded
   • Build an own control plane for Redis that handles restarted pods stuck to an old master address
   • Reduce dependence on the single Redis cluster: move some important data to other stores, such as a separate Redis cluster per server, and use Kafka for some backend components

   Lessons

   The disk resize was not the problem; the cleanup afterwards was. An immutable field pushes operators toward deleting and recreating the controller, and that can set off a full reconcile. Startup and readiness probes are what slow a rolling restart down to a safe pace; without them a distributed store restarts in the most fragile way. Recovery plans must expect a cluster to heal slowly, and dependent services need to degrade gracefully while it does.

   View case
5. IL-0100 PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queries

   PostHog2025

   Symptoms

   On 21 October 2025 at 21:45 UTC a change to timeout handling was deployed to the feature flags service, and two minutes later automated monitoring reported higher error rates. The change was rolled back by 21:49, but error rates stayed high: about 38% of flag evaluation requests in the US datacenter returned errors. At 22:30 Redis metrics showed memory exhaustion on ElastiCache, at 22:35 Postgres connections spiked, and at 22:45 the team found a massive data transfer from Postgres into Redis. The main PostHog application was affected as well, because it shares the Redis instance with the flags service.

   Sustained high node CPUHigh error rateHigh latencyContainer crash loopDatabase overload

   Trigger

   On 21 October a deployment meant to reduce timeout errors (PR #39821); on 24 October the deployment of IP-based rate limiting (PR #40074) that saw all traffic as coming from the load balancer's IP; on 28 October an unrelated routine deployment that rolled the feature flag pods in the US region.

   Root cause

   Capacity exhaustion

   Component

   Node poolPod scheduling and placementIn-memory cacheDatabase

   Contributing factor

   No alert on limitRetry stormIneffective rollbackSingle point of failureNo staged rolloutUnrepresentative staging

   Detection

   On 21 October automated monitoring detected higher error rates two minutes after the deploy. On 24 October no alert existed for 429 errors, and the team learned of the incident from customer reports that began at 18:10 UTC, a 62-minute detection delay. On 28 October on-call received a high error count alert. On 29 October the incident was reported with higher error rates and latency. No alert existed for CPU pressure, so the CPU saturation behind all three incidents was not seen directly.

   Remediation

   • rolled back the timeout handling change within 2 minutes on 21 October, which did not stop the errors
   • throttled connections and Redis writes on 21 October
   • halted the rollout on 28 October, scaled the US flags pods to zero until Redis recovered, then scaled them back gradually
   • rolled back hardware changes on 30 October at 00:03 UTC, which removed most errors but not the latency
   • increased the pod resource requests and adjusted the Kubernetes fleet size so fewer pods ran on each node
   • added retry limits to Envoy between the load balancer and the service
   • moved database connection timeouts and other critical settings to runtime configuration and raised the timeout values
   • added per-node and per-pod CPU alerts at 80% sustained for 5 minutes that page on-call

   Action items

   • Form a strike team from the flags, ingestion and infrastructure teams to review the application and infrastructure for remaining bottlenecks
   • Investigate separating the flags Redis instance from the application Redis instance
   • Before re-enabling ArgoCD sync, align the flags service chart configuration with the values in ArgoCD and define the deployment and Redis strategy
   • Make the feature flags team hard code-owners for flags-related code
   • Build a dashboard of key flag metrics with runbook links and update the feature flag runbooks
   • Add missing alerts on existing service and infrastructure metrics
   • Make readiness checks validate dependencies that degrade under load, for example by pinging the database
   • Rate-limit cache population so it cannot overwhelm Redis
   • Monitor connection pools with automatic throttling near exhaustion and limit concurrent connections
   • Build a production-scale load testing framework
   • Build progressive rollout infrastructure to limit blast radius
   • Document Postgres-to-Redis data flows and write runbooks for data transfer storms

   Lessons

   Nodes that are small relative to pod requests let the scheduler pack many pods on each node, and the resulting CPU saturation showed up three different ways: excessive parallelism, connection pool exhaustion and slow queries. Treating each symptom separately let the problem return; right-sizing the pods and nodes fixed all three. CPU must be alerted on from the start. Unbounded Envoy retries and unbounded cache population amplified the overload, and a Redis instance shared with the main application spread a flags problem to the main application. A rollback did not end the incident, because the data transfer and resource exhaustion it had started kept running.

   View case

By root cause4