Cases2 cases

1 to 2 of 2

1. IL-0470 Buildkite notifications were delayed after Prometheus storage filled and EKS notification workers could not autoscale

   Buildkite2026

   Symptoms

   From 20:01 UTC on 28 May 2026, Buildkite build and job notifications (GitHub commit status, email, Slack and webhook notifications) reached a subset of customers late. Internal queue latency monitoring showed the delay as queues grew while notification-processing workers stayed underprovisioned.

   High latencyAutoscaler misbehaviour

   Trigger

   The Prometheus service used by Buildkite's EKS autoscaling path ran out of storage, so some EKS-based notification workers could not autoscale while their queues were growing.

   Root cause

   Capacity exhaustion

   Component

   Prometheus

   Contributing factor

   No alert on limitPartial rolloutIneffective rollbackUnverified assumptionHarmful manual action

   Detection

   Internal queue latency monitoring detected the rising notification delivery latency.

   Remediation

   • Moved the affected notification-processing workloads from EKS back to the ECS-based infrastructure
   • Adjusted scaling configuration and worker counts by hand
   • Completed the rollback and scaled the remaining affected ECS services
   • Increased Prometheus storage capacity and recorded the change in infrastructure as code

   Action items

   • Alert before Prometheus storage exhaustion can affect autoscaling
   • Fix GitHub notification retry behaviour for a class of errors that could cause repeated retries
   • Make the EKS-to-ECS rollback verify destination capacity, autoscaling configuration and traffic movement
   • Review the CPU, memory and storage allocations of EKS control-plane dependencies such as KEDA and Karpenter
   • Move customer-critical workloads to EKS more gradually, with clearer settling periods
   • Improve customer-level monitoring of notification delivery latency to detect customer-impacting regressions earlier
   • Review which notification types sit on customers' scheduling or CI hot path and need tighter latency, separate queueing or specific alerting
   • Update the status page earlier when notification latency is likely to affect customer workflows
   • Describe the customer-visible impact in status page updates, not only the affected internal service
   • Improve internal escalation for customers who report critical CI impact

   Lessons

   Buildkite said recovery took longer than expected because the rollback path from EKS to ECS did not fully handle this failure, and that its public status page did not show the customer-visible impact as quickly or clearly as it should have, since notification latency affects customers differently depending on how their CI and deployment workflows use notifications.

   View case
2. IL-0182 Too few CoreDNS replicas on EKS were overloaded during traffic spikes, raising latency and cutting the network for task pods, and the over-provisioned, tightly spread CoreDNS pods added later blocked Karpenter consolidation

   이언철2024

   Symptoms

   When traffic changed sharply, CoreDNS came under heavy load, and the cluster saw network problems and a sudden rise in latency. The horizontal pod autoscaler then added pods, database connections surged, and the impact spread to the whole service.

   High latencyIntermittent connection failuresAutoscaler misbehaviour

   Trigger

   Sharp changes in traffic loaded the small number of CoreDNS pods beyond what they could handle.

   Root cause

   Capacity exhaustion

   Component

   CoreDNSKarpenterHorizontal pod autoscalerPod scheduling and placement

   Contributing factor

   Automation amplificationHarmful manual action

   Remediation

   • Set up the Cluster Proportional Autoscaler for CoreDNS with nodesPerReplica and coresPerReplica chosen for the instance types in use
   • Spread CoreDNS with soft topologySpreadConstraints (ScheduleAnyway) instead of anti-affinity so Karpenter could consolidate nodes
   • Added alert thresholds on CoreDNS error rate, panics, request duration and request rate
   • Deployed NodeLocal DNSCache so each node answers DNS from a local cache and forwards misses to the main CoreDNS

   Lessons

   A fixed, small number of CoreDNS pods cannot follow a cluster whose nodes and pods change quickly with traffic, and the damage spreads: DNS latency triggers autoscaling, which in turn floods the database with connections. Simply adding many CoreDNS replicas created a different problem, because with tainted nodes and a maxSkew: 1 spread constraint the pods piled onto a few nodes and blocked Karpenter consolidation. Scaling CoreDNS in proportion to the cluster and putting a DNS cache on every node handled both.

   View case

By root cause1