Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Symptom

Control plane lag

컨트롤 플레인 지연 · control-plane-slow

9 cases · 2019 to 2024

Root cause Capacity exhaustion4 Certificate or credential expiry1 Cloud provider outage1 Configuration change1 Software defect1 Breaking change in an upgrade1

Component Kubernetes API server8 etcd3 Admission webhook2 GKE control plane2

Contributing factor Monitoring blind spot3 Single point of failure3 Unverified assumption2 Limited debug access2

Organisation Civis Analytics1 DFDS1 Google Cloud1 Marcin Cuber1 Namespace

Cases

IL-0084 Packet drops on a bare metal provider's vSwitch forced Namespace to rebuild its fra3 region and left global capacity short, and the next day slow NVMe writes on a control plane node drove the Kubernetes API server to very high CPUNamespace · 2024
Root cause Cloud provider outage
IL-0077 EKS cluster went read-only for over seven hours after Kyverno ephemeralreports filled the etcd database past its size limitMarcin Cuber · 2024 · sev1
Root cause Software defect
Simultaneous cluster auto-upgrades overloaded Scaleway's Kubernetes API, and a node authentication retry loop kept Kapsule and Kosmos nodes NotReady in FR-PAR

View on the graph

Credential broker for pods1

RBAC and service accounts1

Telemetry agent1

No capacity planning2

Automation amplification1

Untested backup1

Configuration not in code1

Unreviewed defaults1

No alert on limit1

No configuration tests1

No staged rollout1

Overlapping failures1

Unrepresentative staging1

1

IL-0168

Scaleway · 2024

Root cause Capacity exhaustion

IL-0060 Kubernetes 1.24 upgrade removed the master node label that Calico route reflectors selected on, dropping all pod routes and taking Reddit down for 314 minutesReddit · 2023

Root cause Breaking change in an upgrade

IL-0140 The GKE API in us-central1 returned errors and high latency for 3 hours 19 minutes after an unexpected rise in backend traffic that client retries made worseGoogle Cloud · 2022

Root cause Capacity exhaustion

IL-0113 The managed GKE control plane of Webhook Relay's main cluster failed because its API server certificates had never been rotated, and a manual credential rotation reported success without taking effectWebhook Relay · 2022

Root cause Certificate or credential expiry

IL-0108 An Alcide admission controller applied to the production cluster during a workshop made create and update requests on Deployments time outDFDS · 2020

Root cause Configuration change

IL-0002 CI builder pods could not get AWS IAM credentials because kubelet fell minutes behind its API rate limitZalando · 2019

Root cause Capacity exhaustion

IL-0075 Kubernetes API servers pegged on CPU when the cluster grew to about 290 nodes, traced to Datadog agents polling the APICivis Analytics · 2019

Root cause Capacity exhaustion