Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Component

Configuration templating

설정 템플릿 · config-templating

5 cases · 2019 to 2025

Root cause Configuration change4 Software defect1

Contributing factor Unverified assumption2 Automation amplification2 Excessive blast radius2 Ignored early signal2 No safeguard on destructive change

Symptom Service outage4 Resources deleted3 OOMKilled2 Load imbalance2

Organisation Grafana Labs2 Skyscanner2 InfluxData1

Cases, by severity

IL-0192 A Crossplane composition merge-policy change for a TLS policy update switched Kubernetes Services from LoadBalancer to ClusterIP and deleted the load balancers behind about 25% of Grafana Cloud servicesGrafana Labs · 2025
Root cause Configuration change
IL-0191 A label selector accidentally added to the Loki distributor Service matched only the new pods of a rollout, which took all write traffic, were OOM-killed and stalled the rolloutGrafana Labs · 2022
Root cause Configuration change

View on the graph

2

No staged rollout2

Missed in review2

Unrepresentative staging2

Untested backup1

Unreviewed defaults1

Monitoring blind spot1

No configuration tests1

High error rate1

Intermittent connection failures1

Probe failures1

IL-0009 A duplicated generated name made Argo CD replace the InfluxDB Cloud production Application and delete the whole production environment

InfluxData · 2022

Root cause Configuration change

IL-0010 A templating change that dropped its {{ }} delimiters made Argo CD delete all 478 Skyscanner services in every regionSkyscanner · 2021

Root cause Configuration change

IL-0011 HAProxy ingress piled requests onto the same pods because a Go template visited the backend map in sorted pod IP orderSkyscanner · 2019

Root cause Software defect