IL-0102 Buildkite · 2026-08-25kubernetesaws
Buildkite had spent four months moving its production workloads from AWS ECS to an AWS EKS cluster, and the cluster had grown steadily throughout. Services in it locate databases, queues and other components through CoreDNS, which ran as a fixed set of three pods and did not scale with the size or rate of change of the cluster. Buildkite had also recently moved time-sensitive notification jobs from the general background worker pool into a new low-latency pool, and had given both pools the same high maximum replica count as the original shared pool, planning to lower both later.
At 22:44 UTC an application deploy consumed the headroom on the deployed nodes. Background workers, including the notification workers, were trying to scale out at the same moment. Their new compute was delayed while the cluster scaled, and since the metrics that had made them scale had not changed, they requested still more pods. The services' maximums would normally have capped this runaway scaling, but they had been set higher than usual. The flood of new pods, endpoints and nodes drove CoreDNS query time from under 1 ms to about 780 ms; queued queries filled memory, all three pods exceeded their memory limits and were restarted, and retries kept them from recovering. With DNS failing, APIs, job dispatch and notifications failed for all customers from 22:47 to 23:16 UTC.
The post-incident analysis found that a defect in a key monitoring query had hidden an upward trend in CoreDNS query duration that tracked the growing cluster size, and that DNS query volume, which had been rising as more workloads moved to EKS, was not monitored well enough. On the client side, recent application performance gains offset the slower DNS answers, so the trend drew no attention, and the planned work to autoscale CoreDNS had not been given priority. Engineers paused deploys, added CoreDNS replicas with more memory and enlarged the node pool that runs them. The new CoreDNS pods were serving by 23:12 UTC, DNS errors dropped quickly, and the accumulated backlog was processed by 23:16 UTC.
From 22:47 UTC on 25 August 2026 the Buildkite web interface, REST API, Agent API, job queue and notifications stopped working for every customer. No one could create a build from the UI or the REST API, job dispatch was delayed, and some builds that were already running when the outage began stayed stuck until the services recovered.
Webhooks from source control providers kept arriving, but their processing fell behind. A subset of customers whose database queries went through PgBouncer instances inside the EKS cluster got error responses from the web UI and on webhook delivery; these errors were less than 1.5% of all requests during the outage. A status post at 23:38 UTC put the failure down to Buildkite's internal DNS services.
Inside the cluster, CoreDNS kept completing queries at a steady rate, but the time to process each query rose from under 1 ms to about 780 ms. Pending queries piled up in memory until all three CoreDNS pods went over their memory limits and Kubernetes restarted them. Continued demand and DNS retries kept CoreDNS from recovering after the restarts.
Root cause Capacity exhaustion
The first status post, at 22:55 UTC, eight minutes after the outage began, said something had gone wrong, and at 23:05 UTC the status page said a sitewide outage was being investigated. CoreDNS query duration had been rising as the cluster grew, but a defect in a key monitoring query and too little monitoring of query volume meant this went unnoticed before the outage.
A CoreDNS deployment sized once and left at a fixed size can carry a growing cluster for months and then fail on the first burst of churn it cannot absorb. When query processing slows, pending queries take up memory, the pods hit their limits and restart, and client retries keep the service down after the restart.
Server-side DNS query duration and volume need their own alerts: here client-side latency was masked by unrelated application speed-ups, and a broken monitoring query hid the server-side trend. Maximum replica counts raised as a temporary measure remove the cap that normally stops runaway scaling when new nodes are slow to arrive.
IL-0027 All CoreDNS pods landed on one node, the node ran out of memory, and DNS for the whole service went downBackpackr · 2025same root cause · shared Container memory limit, CoreDNS, Node pool
IL-0155 Some DOKS clusters in NYC1 saw intermittent DNS timeouts and NodeNotReady events because CoreDNS ran on shared-CPU Droplet worker nodesDigitalOcean · 2026same root cause · shared CoreDNS, Node pool
IL-0082 Losing one CoreDNS pod during a planned EKS node reroll overloaded the remaining CoreDNS pods, causing DNS timeouts and errors across multiple services at RebelMouseRebelMouse · 2025same root cause · shared CoreDNS, Node pool
IL-0099 User servers on 2i2c's Earthscope hub took over 15 minutes to start and kernels were killed during a workshop, because a profile option guaranteed 8 GB of memory with no limit on 32 GB nodes2i2c · 2025same root cause · shared Container memory limit, Node pool
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.
same root cause · shared CoreDNS, Horizontal pod autoscaler
IL-0016 Prometheus on the live cluster ran out of memory and was OOMKilled, and the monitoring node it ran on went NotReadyUK Ministry of Justice · 2023same root cause · shared Container memory limit, Node pool
IL-0094 Student pods filled the shared nodes of a JupyterHub cluster and starved the hub's core pods, freezing the hub2i2c · 2020same root cause · shared Container memory limit, Node pool
IL-0001 Cluster-wide DNS outage after CoreDNS pods were OOM-killed again and again under a burst of DNS queriesZalando · 2019same root cause · shared Container memory limit, CoreDNS
IL-0181 A traffic spike from an external community post pushed product API pods into CPU throttling under a 1-core limit, and an e-commerce product page slowed down while every pod stayed RunningKevin · 2026same root cause · shared Horizontal pod autoscaler
IL-0100 PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queriesPostHog · 2025same root cause · shared Node pool
IL-0048 DNS lookups timed out in a large EKS cluster when CoreDNS hit the 1024 packets-per-second link-local limit, driven by sudo hostname lookupsunknown · 2025same root cause · shared CoreDNS
IL-0204 An etcd memory spike overloaded the control plane of a Render Frankfurt cluster, and DNS then failed because CoreDNS ran on the rebooted control plane nodes and could not start without the API serverRender · 2024same root cause · shared CoreDNS
IL-0044 Prometheus OOMKilled in a loop while replaying its write-ahead log after a kube-prometheus-stack upgradeSaifeddine Rajhi · 2024same root cause · shared Container memory limit
IL-0110 A Grafana Agent pod's high memory use and crash-looping Hangfire pods drove a Kubernetes node into system OOM and NotReady, twice in one dayDFDS · 2024same root cause · shared Container memory limit
IL-0169 The Scaleway Kubernetes API in FR-PAR ran out of memory under simultaneous deployments, and because it also handled node authentication, Kapsule and Kosmos nodes went NotReadyScaleway · 2024same root cause · shared Container memory limit
IL-0165 Some LKE users in Frankfurt could not reach their kube-apiservers with kubectl after backend services failed under memory constraintsLinode · 2023same root cause · shared Container memory limit
IL-0070 Critical pods on core nodes evicted when Prometheus began scraping about 200 nodes after a dask-gateway scale-up2i2c · 2023same root cause · shared Node pool
IL-0206 Consul agents in GitLab.com's GKE pods were repeatedly OOM-killed, so the Rails pods lost database replica discovery and sent every query to the primaryGitLab · 2022same root cause · shared Container memory limit
IL-0013 The conntrack table filled up on downsized GKE nodes and a high-traffic aggregation proxy saw climbing connection resetsloveholidays · 2020same root cause · shared Node pool
IL-0015 A GKE cluster's /16 pod address range ran out at 256 nodes, leaving new pods Pending and node scale-ups failingloveholidays · 2020same root cause · shared Node pool
IL-0026 Kubernetes nodes hit SystemOOM because fluentd-sumologic log shipper pods had no memory limitsBlue Matador · 2019same root cause · shared Container memory limit