Cases45 cases

1 to 5 of 45

1. IL-0102 CoreDNS pods on a production EKS cluster ran out of memory under a burst of new pods, endpoints and nodes, taking down the Buildkite web UI, APIs and job dispatch

   Buildkite2026

   Symptoms

   From 22:47 UTC on 25 August 2026 the Buildkite web interface, REST API, Agent API, job queue and notifications stopped working for every customer. No one could create a build from the UI or the REST API, job dispatch was delayed, and some builds that were already running when the outage began stayed stuck until the services recovered.

   Service outageDNS resolution failuresOOMKilledAutoscaler misbehaviour

   Trigger

   At 22:44 UTC an application deploy created a surge of new pods, which used up the free capacity on the existing nodes and made the EKS cluster scale out. Background worker pools with unusually high maximum replica counts were scaling out at the same time and kept asking for more pods while new nodes were still being provisioned. Together they produced an unusually high rate of change in applications, network endpoints and nodes, and with it a sudden rise in demand on CoreDNS.

   Root cause

   Capacity exhaustion

   Component

   CoreDNSContainer memory limitHorizontal pod autoscalerNode pool

   Contributing factor

   No headroomAutomation amplificationRetry stormMonitoring blind spot

   Detection

   The first status post, at 22:55 UTC, eight minutes after the outage began, said something had gone wrong, and at 23:05 UTC the status page said a sitewide outage was being investigated. CoreDNS query duration had been rising as the cluster grew, but a defect in a key monitoring query and too little monitoring of query volume meant this went unnoticed before the outage.

   Remediation

   • paused further application deploys
   • scaled CoreDNS from three to twelve pods
   • raised each CoreDNS pod's memory limit from 512 MiB to 2 GiB
   • expanded the system node pool that runs CoreDNS

   Action items

   • Increase immediate CoreDNS headroom: twelve pods instead of three, a 2 GiB memory limit instead of 512 MiB, and a larger system node pool
   • Enable EKS-managed CoreDNS autoscaling with a tested minimum replica count and enough capacity to spread the replicas across nodes and availability zones
   • Add direct alerts on CoreDNS query duration, query volume, goroutine growth, memory pressure, OOM restarts and available replicas, and track longer-term trends in capacity planning
   • Find other cluster-critical services that lack tested capacity, direct alerting, spread across failure domains, and either safe autoscaling or documented static headroom, and fix them

   Lessons

   A CoreDNS deployment sized once and left at a fixed size can carry a growing cluster for months and then fail on the first burst of churn it cannot absorb. When query processing slows, pending queries take up memory, the pods hit their limits and restart, and client retries keep the service down after the restart. Server-side DNS query duration and volume need their own alerts: here client-side latency was masked by unrelated application speed-ups, and a broken monitoring query hid the server-side trend. Maximum replica counts raised as a temporary measure remove the cap that normally stops runaway scaling when new nodes are slow to arrive.

   View case
2. IL-0155 Some DOKS clusters in NYC1 saw intermittent DNS timeouts and NodeNotReady events because CoreDNS ran on shared-CPU Droplet worker nodes

   DigitalOcean2026

   Symptoms

   From 20:31 UTC on July 9, 2026, DigitalOcean reported a problem with Kubernetes deployments in its NYC1 region. Application workloads could see intermittent DNS failures and NodeNotReady events.

   DNS resolution failuresIntermittent timeoutsNode NotReady

   Root cause

   Capacity exhaustion

   Component

   CoreDNSNode pool

   Contributing factor

   Unknown limit

   Lessons

   Cluster DNS is sensitive to latency, and CoreDNS on shared-CPU nodes can time out intermittently. DigitalOcean's advice is to place CoreDNS on dedicated-CPU node pools and to run enough CoreDNS replicas.

   View case
3. IL-0237 A Kubernetes patch upgrade evicted every Ray Serve head and worker pod at once, and Lucidworks hosted LLMs went down

   Lucidworks2026sev1

   Symptoms

   From 16:18 UTC on 28 May 2026 the Lucidworks AI hosted LLMs in the us-southcarolina region were unavailable: queries to llama-3-8b-instruct, llama-3v2-3b-instruct and phi-4-multimodal-instruct returned 500 or 429 errors. Search, embedding models, the Lucidworks Platform UI and passthrough LLM calls kept working.

   Service outagePod evictionImage pull failures

   Trigger

   A routine Kubernetes patch upgrade on a cluster in us-southcarolina cycled its node pools one after another in quick succession.

   Root cause

   Faulty deployment

   Component

   Node poolModel inference server

   Contributing factor

   No staged rolloutOverlapping failures

   Remediation

   • Deleted the degraded replacement head node that could not pull container images
   • Waited for the LLM image pulls to finish on the replacement nodes
   • Launched new nodes with the required LLM images and redirected Lucidworks AI traffic to them

   Action items

   • Upgrade clusters that host LLM workloads in sequence, with each node pool fully healthy before the next one is upgraded
   • Investigate image pre-loading, such as pre-baked disks or image streaming, for large ML model images
   • Open a support ticket with the cloud provider about the abnormal Docker pull times
   • Coordinate Kubernetes maintenance windows with LLM service owners so they do not clash with ongoing deployments
   • Build tooling to force a Ray cluster failover to the blue or green state instead of waiting for Ray to settle it

   View case
4. IL-0548 Buildkite's EKS Prometheus server was killed for memory again and again, so services stopped scaling and jobs lagged

   Buildkite2026

   Symptoms

   On 20 May 2026 a subset of Buildkite customers saw delays in notification delivery, build dispatch and the metrics services. The status page first reported delays to notifications for all customers and later narrowed it to a subset of customers.

   High latencyOOMKilledMonitoring outageAutoscaler misbehaviour

   Trigger

   Autoscaling during Buildkite's migration of production services from AWS Fargate to AWS EKS raised the number of pods that Prometheus tracked, and with it the memory the Prometheus server needed.

   Root cause

   Capacity exhaustion

   Component

   PrometheusContainer memory limitNode pool

   Contributing factor

   No resource requests or limits

   Remediation

   • Moved the load of the affected services back to AWS Fargate by following prepared documentation
   • Scaled up the services of a few customers with very large backlogs further by hand

   Action items

   • Upsize the underlying system nodes
   • Set higher requests and limits for the Prometheus server so it can handle more load
   • Review all new EKS resources and set any missing requests and limits
   • Add observability and monitors for EKS pod and node health

   View case
5. IL-0166 A compute host update removed the credential parser used to decode StackScripts, so new Linodes and every new LKE and LKE-E node failed to boot and autoscaling and node recycling failed in all regions

   Linode2026

   Symptoms

   Linodes deployed with StackScripts failed to deploy and could not boot. Every newly created Linode Kubernetes Engine (LKE) and LKE-E node relies on StackScripts, so LKE autoscaling and node recycling failed as well, in all regions.

   Scale-up failure

   Trigger

   A recent update on the compute hosts removed a credential parser that is used when decoding StackScripts.

   Root cause

   Faulty deployment

   Component

   Cluster autoscalerNode pool

   Contributing factor

   –

   Detection

   The issue was identified at 18:18 UTC on 2026-05-15. The first status post went up at 00:42 UTC on 2026-05-16.

   Remediation

   • tested and applied a rollback starting at 01:10 UTC on 2026-05-16, and failures dropped from 01:20 UTC

   Lessons

   Removing a credential parser from the compute hosts broke the boot of every Linode deployed with StackScripts, and every new LKE and LKE-E node is one of them, so node creation, autoscaling and node recycling stopped in all regions. The rollback reduced failures ten minutes after testing began.

   View case

By root cause10