Cases45 cases

1 to 5 of 45

1. IL-0471 GitHub Actions runs failed or sat queued for 9 hours after a routine deployment's pod replacement saturated capacity

   GitHub2026

   Symptoms

   From 15:05 UTC on 6 August 2026 GitHub Actions workflow runs failed to start, failed partway through or stayed queued for a long time, on both GitHub-hosted and self-hosted runners. Some requests to the Actions REST API returned errors, and some customers saw unexpected rate limiting in their workflows.

   High error rateHigh latency

   Trigger

   A routine deployment to an internal Actions service that processes events and generates Actions jobs replaced pods, and the remaining capacity became saturated.

   Root cause

   Capacity exhaustion

   Component

   CI system

   Contributing factor

   No headroomExcessive blast radiusHarmful manual action

   Remediation

   • Expanded capacity of the affected services
   • Throttled incoming webhook-triggered work
   • Increased processing capacity for the backlog of affected events
   • Deployed changes that stopped runners from repeatedly trying to acquire invalid jobs
   • Deployed a fix for self-hosted runners that were not picking up jobs and enabled it incrementally
   • Paused migrations with GitHub Enterprise Importer
   • Restored webhook-triggered Actions workflows to full throughput
   • Rolled back the mitigation that left some ARC runners offline, and recovered those runners by hand
   • Shared CLI and UI steps for customers to clear jobs that could not be retried or canceled

   Action items

   • Improve deployment and capacity safeguards for the affected services
   • Strengthen monitoring for the conditions that preceded the incident
   • Improve the resiliency and recovery of queued work and runner assignment
   • Add automatic recovery for self-hosted runners in upcoming Runner and ARC releases
   • Reduce the risk of cascading failures and speed up recovery during large Actions disruptions

   View case
2. IL-0470 Buildkite notifications were delayed after Prometheus storage filled and EKS notification workers could not autoscale

   Buildkite2026

   Symptoms

   From 20:01 UTC on 28 May 2026, Buildkite build and job notifications (GitHub commit status, email, Slack and webhook notifications) reached a subset of customers late. Internal queue latency monitoring showed the delay as queues grew while notification-processing workers stayed underprovisioned.

   High latencyAutoscaler misbehaviour

   Trigger

   The Prometheus service used by Buildkite's EKS autoscaling path ran out of storage, so some EKS-based notification workers could not autoscale while their queues were growing.

   Root cause

   Capacity exhaustion

   Component

   Prometheus

   Contributing factor

   No alert on limitPartial rolloutIneffective rollbackUnverified assumptionHarmful manual action

   Detection

   Internal queue latency monitoring detected the rising notification delivery latency.

   Remediation

   • Moved the affected notification-processing workloads from EKS back to the ECS-based infrastructure
   • Adjusted scaling configuration and worker counts by hand
   • Completed the rollback and scaled the remaining affected ECS services
   • Increased Prometheus storage capacity and recorded the change in infrastructure as code

   Action items

   • Alert before Prometheus storage exhaustion can affect autoscaling
   • Fix GitHub notification retry behaviour for a class of errors that could cause repeated retries
   • Make the EKS-to-ECS rollback verify destination capacity, autoscaling configuration and traffic movement
   • Review the CPU, memory and storage allocations of EKS control-plane dependencies such as KEDA and Karpenter
   • Move customer-critical workloads to EKS more gradually, with clearer settling periods
   • Improve customer-level monitoring of notification delivery latency to detect customer-impacting regressions earlier
   • Review which notification types sit on customers' scheduling or CI hot path and need tighter latency, separate queueing or specific alerting
   • Update the status page earlier when notification latency is likely to affect customer workflows
   • Describe the customer-visible impact in status page updates, not only the affected internal service
   • Improve internal escalation for customers who report critical CI impact

   Lessons

   Buildkite said recovery took longer than expected because the rollback path from EKS to ECS did not fully handle this failure, and that its public status page did not show the customer-visible impact as quickly or clearly as it should have, since notification latency affects customers differently depending on how their CI and deployment workflows use notifications.

   View case
3. IL-0548 Buildkite's EKS Prometheus server was killed for memory again and again, so services stopped scaling and jobs lagged

   Buildkite2026

   Symptoms

   On 20 May 2026 a subset of Buildkite customers saw delays in notification delivery, build dispatch and the metrics services. The status page first reported delays to notifications for all customers and later narrowed it to a subset of customers.

   High latencyOOMKilledMonitoring outageAutoscaler misbehaviour

   Trigger

   Autoscaling during Buildkite's migration of production services from AWS Fargate to AWS EKS raised the number of pods that Prometheus tracked, and with it the memory the Prometheus server needed.

   Root cause

   Capacity exhaustion

   Component

   PrometheusContainer memory limitNode pool

   Contributing factor

   No resource requests or limits

   Remediation

   • Moved the load of the affected services back to AWS Fargate by following prepared documentation
   • Scaled up the services of a few customers with very large backlogs further by hand

   Action items

   • Upsize the underlying system nodes
   • Set higher requests and limits for the Prometheus server so it can handle more load
   • Review all new EKS resources and set any missing requests and limits
   • Add observability and monitors for EKS pod and node health

   View case
4. IL-0181 A traffic spike from an external community post pushed product API pods into CPU throttling under a 1-core limit, and an e-commerce product page slowed down while every pod stayed Running

   Kevin2026

   Symptoms

   An e-commerce platform suddenly became slow to respond, worst on the product lookup page, and users began reporting that the page would not open.

   CPU throttlingHigh latency

   Trigger

   A post on the external community Ppomppu linked to the platform and brought a large number of users in a short time, who called the product lookup API heavily.

   Root cause

   Capacity exhaustion

   Component

   Container CPU limitHorizontal pod autoscaler

   Contributing factor

   Monitoring blind spot

   Detection

   Users reported that the product page would not open. The dashboard showed all pods Running, and the namespace CPU usage near 100% and the CPU usage metrics pointed to throttling.

   Remediation

   • raised the HPA minReplicas from 5 to 10
   • raised the container CPU limit from 1 core to 2 cores

   Action items

   • Keep adjusting the CPU limit to a suitable level through monitoring

   Lessons

   CPU throttling is hard to notice: pods stay Running, nothing restarts and the logs show no clear error, so a pod list does not reveal it and CPU usage metrics do. Settings that had handled expected event traffic became the bottleneck as soon as one external post multiplied the traffic. The author concludes that the team needs more preparation for traffic pattern changes and more thought about resource settings.

   View case
5. IL-0063 VPN outage after recreating a Redis StatefulSet restarted the whole Redis cluster without readiness checks

   AdGuard VPN2025

   Symptoms

   The day began with an alert that several instances of the Redis cluster were running out of disk space. That part was fixed quickly. The outage came an hour later: right after the Redis StatefulSet was deleted and recreated at 8:10 UTC, Kubernetes restarted every Redis pod within about five minutes, and problems started to appear.

   Service outageOOMKilledProbe failuresHigh latency

   Trigger

   To make Git match the volumes that had been resized by hand, the Redis StatefulSet was deleted with orphan (non-cascading) deletion and recreated with the new storage template; Kubernetes reconciled the new spec by restarting the entire Redis cluster.

   Root cause

   Configuration change

   Component

   Persistent volumes and stateful setsIn-memory cacheContainer memory limit

   Contributing factor

   No health probesUnverified assumptionSingle point of failureRetry stormNo headroom

   Detection

   An alert about Redis instances running out of disk space started the work; the outage itself showed up as backend errors about Redis connections right after the StatefulSet was recreated.

   Remediation

   • made the Redis masters forget obsolete node identities, reset the affected replicas and rejoined them, then waited for replication to catch up
   • identified the component that was actually running out of memory, scaled the backend service and raised memory limits for that component
   • temporarily relaxed probes so backend instances took traffic immediately instead of flapping between ready and not ready
   • reverted the temporary workarounds at 10:00 UTC

   Action items

   • Add startup probes so a Redis pod is not marked ready until its dataset is loaded
   • Build an own control plane for Redis that handles restarted pods stuck to an old master address
   • Reduce dependence on the single Redis cluster: move some important data to other stores, such as a separate Redis cluster per server, and use Kafka for some backend components

   Lessons

   The disk resize was not the problem; the cleanup afterwards was. An immutable field pushes operators toward deleting and recreating the controller, and that can set off a full reconcile. Startup and readiness probes are what slow a rolling restart down to a safe pace; without them a distributed store restarts in the most fragile way. Recovery plans must expect a cluster to heal slowly, and dependent services need to degrade gracefully while it does.

   View case

By root cause7