Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Component

Credential broker for pods

파드용 자격 증명 중개자 · credential-broker

5 cases · 2019 to 2023

Root cause Capacity exhaustion1 Configuration change1 Dependency failure1 Operator error1 Software defect1

Contributing factor Unreviewed defaults2 Single point of failure2 Automation amplification1 Excessive blast radius1

Symptom High latency3 Credentials unavailable2 Control plane lag1 High error rate1

Organisation Adevinta1 Google Cloud1 Target1 UK Ministry of Justice1 Zalando

Cases, by severity

IL-0073 Intermittent Kafka connectivity overloaded Docker daemons, set off a node rescheduling cascade and poisoned the Consul gossip meshTarget · 2019 · sev1
Root cause Dependency failure
IL-0141 gke-metadata-server versions 0.4.272 to 0.4.280 flooded Cloud Logging with "Unable to sync sandbox" debug logs on GKE clusters with Workload Identity, using up logging quota and adding billable usageGoogle Cloud · 2023
Root cause Software defect
A test cluster deletion script run with the kube context pointing at the live-1 production cluster deleted its ingress controller, monitoring, logging, cert-manager, kiam and external-dns

View on the graph

Unpatched known bug1

No safeguard on destructive change1

Monitoring outage1

Resources deleted1

Service outage1

Stale endpoint routing1

Load imbalance1

1

IL-0089

UK Ministry of Justice · 2020

Root cause Operator error

IL-0189 A JVM service moved from EC2 to Kubernetes answered up to ten times slower because KIAM's 15-minute credentials made the AWS Java SDK fetch new credentials on every requestAdevinta · 2019

Root cause Configuration change

IL-0002 CI builder pods could not get AWS IAM credentials because kubelet fell minutes behind its API rate limitZalando · 2019

Root cause Capacity exhaustion