Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Contributing factor

No configuration tests

설정 테스트 부재 · no-config-tests

7 cases · 2020 to 2023

Root cause Configuration change6 Software defect1

Component Node pool3 AWS VPC CNI3 Cloud load balancer2 Node launch template2 Terraform2

Symptom Service outage4 High error rate2 IP address exhaustion2 Pods stuck in ContainerCreating2

Organisation DFDS4 UK Ministry of Justice2 Skyscanner1

Cases

IL-0019 An EKS Terraform module upgrade shrank node root disks from 100 GB to 20 GB and image pulls failed with no space left on deviceUK Ministry of Justice · 2023
Root cause Configuration change
IL-0017 AWS VPC CNI stopped allocating pod IP addresses on six nodes of the live cluster, leaving new pods in ContainerCreatingUK Ministry of Justice · 2023
Root cause Configuration change
IL-0109 A retried node rollover on a DFDS EKS cluster set the Auto Scaling group back to 15 instances, and the default termination policy removed every healthy worker node, leaving the cluster with no nodes

View on the graph

Admission webhook1

Configuration templating1

Container runtime1

Ingress controller1

Kubernetes API server1

Node root disk1

Telemetry agent1

Resources deleted2

Control plane lag1

Deploys failing1

Image pull failures1

Monitoring outage1

Node disk full1

Probe failures1

DFDS · 2023 · sev1

Root cause Configuration change

IL-0106 A malformed Traefik IngressRoute triggered a Traefik v2 bug once a node rollover restarted every Traefik pod, and all Kubernetes services returned 404DFDS · 2021

Root cause Software defect

IL-0010 A templating change that dropped its {{ }} delimiters made Argo CD delete all 478 Skyscanner services in every regionSkyscanner · 2021

Root cause Configuration change

IL-0105 Replacing the AWS VPC CNI during a Terraform refactoring detached the ALB target groups, and every service in the Kubernetes cluster returned 503DFDS · 2021

Root cause Configuration change

IL-0108 An Alcide admission controller applied to the production cluster during a workshop made create and update requests on Deployments time outDFDS · 2020

Root cause Configuration change