Cases8 cases

3 to 7 of 8

1. IL-0548 Buildkite's EKS Prometheus server was killed for memory again and again, so services stopped scaling and jobs lagged

   Buildkite2026

   Symptoms

   On 20 May 2026 a subset of Buildkite customers saw delays in notification delivery, build dispatch and the metrics services. The status page first reported delays to notifications for all customers and later narrowed it to a subset of customers.

   High latencyOOMKilledMonitoring outageAutoscaler misbehaviour

   Trigger

   Autoscaling during Buildkite's migration of production services from AWS Fargate to AWS EKS raised the number of pods that Prometheus tracked, and with it the memory the Prometheus server needed.

   Root cause

   Capacity exhaustion

   Component

   PrometheusContainer memory limitNode pool

   Contributing factor

   No resource requests or limits

   Remediation

   • Moved the load of the affected services back to AWS Fargate by following prepared documentation
   • Scaled up the services of a few customers with very large backlogs further by hand

   Action items

   • Upsize the underlying system nodes
   • Set higher requests and limits for the Prometheus server so it can handle more load
   • Review all new EKS resources and set any missing requests and limits
   • Add observability and monitors for EKS pod and node health

   View case
2. IL-0156 Revoking old certificates after a rotation left a certificate key mismatch between API endpoints and the cluster controller, and LKE cluster management and autoscaling in Atlanta returned 5xx errors

   Linode2025

   Symptoms

   From about 02:29 UTC on 24 September 2025, customers of Linode Kubernetes Engine (LKE) in US-Southeast (Atlanta) could not manage their clusters, and autoscaling was impacted. Internal logs showed 5xx errors whenever customers tried to manage LKE clusters associated with the affected cluster controller.

   High error rateAutoscaler misbehaviour

   Trigger

   Old certificates on the LKE cluster controllers were revoked at 02:29 UTC on 24 September 2025, after the expiring certificates had been rotated on 23 September.

   Root cause

   Certificate or credential expiry

   Component

   Managed Kubernetes control plane and provider APICluster autoscaler

   Contributing factor

   Change with a delayed effect

   Remediation

   • deployed a fix for the certificate key mismatch
   • restarted the API endpoints

   Action items

   • Continue investigating the root cause and take preventive actions

   View case
3. IL-0182 Too few CoreDNS replicas on EKS were overloaded during traffic spikes, raising latency and cutting the network for task pods, and the over-provisioned, tightly spread CoreDNS pods added later blocked Karpenter consolidation

   이언철2024

   Symptoms

   When traffic changed sharply, CoreDNS came under heavy load, and the cluster saw network problems and a sudden rise in latency. The horizontal pod autoscaler then added pods, database connections surged, and the impact spread to the whole service.

   High latencyIntermittent connection failuresAutoscaler misbehaviour

   Trigger

   Sharp changes in traffic loaded the small number of CoreDNS pods beyond what they could handle.

   Root cause

   Capacity exhaustion

   Component

   CoreDNSKarpenterHorizontal pod autoscalerPod scheduling and placement

   Contributing factor

   Automation amplificationHarmful manual action

   Remediation

   • Set up the Cluster Proportional Autoscaler for CoreDNS with nodesPerReplica and coresPerReplica chosen for the instance types in use
   • Spread CoreDNS with soft topologySpreadConstraints (ScheduleAnyway) instead of anti-affinity so Karpenter could consolidate nodes
   • Added alert thresholds on CoreDNS error rate, panics, request duration and request rate
   • Deployed NodeLocal DNSCache so each node answers DNS from a local cache and forwards misses to the main CoreDNS

   Lessons

   A fixed, small number of CoreDNS pods cannot follow a cluster whose nodes and pods change quickly with traffic, and the damage spreads: DNS latency triggers autoscaling, which in turn floods the database with connections. Simply adding many CoreDNS replicas created a different problem, because with tainted nodes and a maxSkew: 1 spread constraint the pods piled onto a few nodes and blocked Karpenter consolidation. Scaling CoreDNS in proportion to the cluster and putting a DNS cache on every node handled both.

   View case
4. IL-0163 An API change that slipped through a gap in automated testing made LKE clusters resize to the wrong number of nodes and added billable worker nodes to customer accounts

   Linode2023

   Symptoms

   Resizing an LKE cluster produced an incorrect number of nodes, so some customer clusters were resized in unexpected ways and the extra worker nodes were billed to the customer accounts. The clusters themselves kept functioning. Linode's first status update, at 05:07 UTC on 1 June 2023, advised customers to avoid resizing their clusters if possible.

   Autoscaler misbehaviour

   Trigger

   At about 16:00 UTC on 31 May 2023 a change to the API that caused LKE clusters to scale improperly was pushed to production.

   Root cause

   Faulty deployment

   Component

   Node pool

   Contributing factor

   Gap in automated tests

   Detection

   The first status update, at 05:07 UTC on 1 June, said Linode had been made aware of an issue when resizing LKE clusters, about 13 hours after the change went out.

   Remediation

   • advised customers to avoid resizing their clusters
   • deployed a hotfix at 09:40 UTC on 1 June that corrected the scaling behaviour
   • left the extra worker nodes in place to avoid disrupting customer workloads
   • contacted affected customers by ticket about credit adjustments and further mitigation

   Action items

   • Audit the automated tests
   • Add pre-deployment checklist items so the same failure does not happen again

   Lessons

   An API change that alters scaling behaviour resized customer clusters and added cost while the clusters themselves kept working; Linode announced the issue about 13 hours after the change went out. Removing the extra nodes would have disrupted workloads, so the cost was handled through credits instead.

   View case
5. IL-0209 A too-high HPA CPU target kept GitLab.com web pods from scaling up as traffic moved from VMs to Kubernetes, breaching the web latency SLO for 15 minutes

   GitLab2021

   Symptoms

   The web service breached its latency SLO at 07:30 UTC and its Apdex dropped: some requests to the GitLab.com web front end slowed down for about 15 minutes. This happened while web traffic was being shifted from virtual machines to the web pods on Kubernetes, and the pods did not scale up as the traffic arrived.

   High latencyAutoscaler misbehaviour

   Trigger

   Shifting web traffic from VMs to Kubernetes pods during the GitLab.com web migration.

   Root cause

   Configuration change

   Component

   Horizontal pod autoscaler

   Contributing factor

   Unverified assumption

   Detection

   The web latency SLO breach at 07:30 UTC; an engineer declared the incident in Slack at 07:40.

   Remediation

   • Lowered the target CPU utilisation of the web pods' autoscaler to a more conservative value (merge request gitlab-com/gl-infra/k8s-workloads/gitlab-com!1149)

   Action items

   • Adjust the target utilisation for the web pods (gitlab-com/gl-infra/k8s-workloads/gitlab-com!1149)

   Lessons

   When traffic is moved onto pods that rely on CPU-based autoscaling, a CPU target set too high can keep the pods from scaling as the traffic arrives; GitLab fixed it by setting a more conservative target.

   View case

By root cause5