Cases2 cases

1 to 2 of 2

1. IL-0470 Buildkite notifications were delayed after Prometheus storage filled and EKS notification workers could not autoscale

   Buildkite2026

   Symptoms

   From 20:01 UTC on 28 May 2026, Buildkite build and job notifications (GitHub commit status, email, Slack and webhook notifications) reached a subset of customers late. Internal queue latency monitoring showed the delay as queues grew while notification-processing workers stayed underprovisioned.

   High latencyAutoscaler misbehaviour

   Trigger

   The Prometheus service used by Buildkite's EKS autoscaling path ran out of storage, so some EKS-based notification workers could not autoscale while their queues were growing.

   Root cause

   Capacity exhaustion

   Component

   Prometheus

   Contributing factor

   No alert on limitPartial rolloutIneffective rollbackUnverified assumptionHarmful manual action

   Detection

   Internal queue latency monitoring detected the rising notification delivery latency.

   Remediation

   • Moved the affected notification-processing workloads from EKS back to the ECS-based infrastructure
   • Adjusted scaling configuration and worker counts by hand
   • Completed the rollback and scaled the remaining affected ECS services
   • Increased Prometheus storage capacity and recorded the change in infrastructure as code

   Action items

   • Alert before Prometheus storage exhaustion can affect autoscaling
   • Fix GitHub notification retry behaviour for a class of errors that could cause repeated retries
   • Make the EKS-to-ECS rollback verify destination capacity, autoscaling configuration and traffic movement
   • Review the CPU, memory and storage allocations of EKS control-plane dependencies such as KEDA and Karpenter
   • Move customer-critical workloads to EKS more gradually, with clearer settling periods
   • Improve customer-level monitoring of notification delivery latency to detect customer-impacting regressions earlier
   • Review which notification types sit on customers' scheduling or CI hot path and need tighter latency, separate queueing or specific alerting
   • Update the status page earlier when notification latency is likely to affect customer workflows
   • Describe the customer-visible impact in status page updates, not only the affected internal service
   • Improve internal escalation for customers who report critical CI impact

   Lessons

   Buildkite said recovery took longer than expected because the rollback path from EKS to ECS did not fully handle this failure, and that its public status page did not show the customer-visible impact as quickly or clearly as it should have, since notification latency affects customers differently depending on how their CI and deployment workflows use notifications.

   View case
2. IL-0209 A too-high HPA CPU target kept GitLab.com web pods from scaling up as traffic moved from VMs to Kubernetes, breaching the web latency SLO for 15 minutes

   GitLab2021

   Symptoms

   The web service breached its latency SLO at 07:30 UTC and its Apdex dropped: some requests to the GitLab.com web front end slowed down for about 15 minutes. This happened while web traffic was being shifted from virtual machines to the web pods on Kubernetes, and the pods did not scale up as the traffic arrived.

   High latencyAutoscaler misbehaviour

   Trigger

   Shifting web traffic from VMs to Kubernetes pods during the GitLab.com web migration.

   Root cause

   Configuration change

   Component

   Horizontal pod autoscaler

   Contributing factor

   Unverified assumption

   Detection

   The web latency SLO breach at 07:30 UTC; an engineer declared the incident in Slack at 07:40.

   Remediation

   • Lowered the target CPU utilisation of the web pods' autoscaler to a more conservative value (merge request gitlab-com/gl-infra/k8s-workloads/gitlab-com!1149)

   Action items

   • Adjust the target utilisation for the web pods (gitlab-com/gl-infra/k8s-workloads/gitlab-com!1149)

   Lessons

   When traffic is moved onto pods that rely on CPU-based autoscaling, a CPU target set too high can keep the pods from scaling as the traffic arrives; GitLab fixed it by setting a more conservative target.

   View case

By root cause2