Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Component

CI system

CI 시스템 · ci-system

3 cases · 2019 to 2023

Root cause Capacity exhaustion1 Configuration change1 Operator error1

Contributing factor Unverified assumption1 Credential in configuration1 Unreviewed defaults1 No safeguard on destructive change1

Symptom Control plane lag1 Credentials unavailable1 High error rate1 Image pull failures1

Organisation Spotify1 UK Ministry of Justice1 Zalando1

Cases, by severity

IL-0018 A Docker Hub password rotation left EKS nodes and Concourse locked out of Docker Hub and unable to pull imagesUK Ministry of Justice · 2023
Root cause Operator error
IL-0002 CI builder pods could not get AWS IAM credentials because kubelet fell minutes behind its API rate limitZalando · 2019
Root cause Capacity exhaustion
IL-0061 A Terraform pull request merged out of order deleted a production GKE cluster, and recreating it deleted a second one

View on the graph

Missed in review1

Shared credential1

Single point of failure1

Resources deleted1

Spotify · 2019

Root cause Configuration change