Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Component

Node launch template

노드 런치 템플릿 · launch-template

4 cases · 2022 to 2023

Root cause Configuration change2 Operator error1 Software defect1

Contributing factor Automation amplification2 No configuration tests2 Credential in configuration1 Unreviewed defaults1

Symptom Image pull failures2 Intermittent connection failures1 Intermittent timeouts1 Monitoring outage1

Organisation DFDS2 UK Ministry of Justice2

Cases, by severity

IL-0109 A retried node rollover on a DFDS EKS cluster set the Auto Scaling group back to 15 instances, and the default termination policy removed every healthy worker node, leaving the cluster with no nodesDFDS · 2023 · sev1
Root cause Configuration change
IL-0019 An EKS Terraform module upgrade shrank node root disks from 100 GB to 20 GB and image pulls failed with no space left on deviceUK Ministry of Justice · 2023
Root cause Configuration change
A Docker Hub password rotation left EKS nodes and Concourse locked out of Docker Hub and unable to pull images

View on the graph

Harmful manual action1

Monitoring blind spot1

No alert on limit1

No health probes1

No staged rollout1

Overlapping failures1

Missed in review1

Shared credential1

Unrepresentative staging1

IP address exhaustion1

Node disk full1

Pods stuck in ContainerCreating1

Resources deleted1

Service outage1

IL-0018

UK Ministry of Justice · 2023

Root cause Operator error

IL-0112 The daily node rollover put DFDS's EKS nodes on an AMI later recalled for a kernel issue, causing "PLEG is not healthy" NodeNotReady flaps and pod network timeoutsDFDS · 2022

Root cause Software defect