Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Symptom

Autoscaler misbehaviour

오토스케일러 오동작 · autoscaler-misbehaving

6 cases · 2019 to 2026

Root cause Capacity exhaustion2 Certificate or credential expiry1 Cloud provider outage1 Configuration change1 Faulty deployment1

Component Horizontal pod autoscaler4 Node pool3 Cluster autoscaler2 CoreDNS2

Contributing factor Automation amplification2 Unverified assumption1 Change with a delayed effect1 Ignored early signal1

Organisation Linode2 Buildkite1 GitLab1 loveholidays1 이언철1

Cases

IL-0102 CoreDNS pods on a production EKS cluster ran out of memory under a burst of new pods, endpoints and nodes, taking down the Buildkite web UI, APIs and job dispatchBuildkite · 2026
Root cause Capacity exhaustion
IL-0156 Revoking old certificates after a rotation left a certificate key mismatch between API endpoints and the cluster controller, and LKE cluster management and autoscaling in Atlanta returned 5xx errorsLinode · 2025
Root cause Certificate or credential expiry
Too few CoreDNS replicas on EKS were overloaded during traffic spikes, raising latency and cutting the network for task pods, and the over-provisioned, tightly spread CoreDNS pods added later blocked Karpenter consolidation

View on the graph

GKE control plane1

Ingress controller1

Managed Kubernetes control plane and provider API1

Container memory limit1

Pod scheduling and placement1

Harmful manual action1

Monitoring blind spot1

Overlapping failures1

Unrepresentative staging1

Gap in automated tests1

IL-0182

이언철 · 2024

Root cause Capacity exhaustion

IL-0163 An API change that slipped through a gap in automated testing made LKE clusters resize to the wrong number of nodes and added billable worker nodes to customer accountsLinode · 2023

Root cause Faulty deployment

IL-0209 A too-high HPA CPU target kept GitLab.com web pods from scaling up as traffic moved from VMs to Kubernetes, breaching the web latency SLO for 15 minutesGitLab · 2021

Root cause Configuration change

IL-0014 A GKE upgrade to a withdrawn version stalled on slow drains and a stuck node pool, then broke an ingress and confused the HPAloveholidays · 2019

Root cause Cloud provider outage