Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Root cause

Cloud provider outage

클라우드 제공자 장애 · cloud-provider

8 cases · 2019 to 2025

Component Application database4 Cluster autoscaler3 Node pool3 AKS control plane1 GKE control plane1 Helm1 Horizontal pod autoscaler1 Ingress controller1 Kubernetes API server1 Node root disk1 Persistent volumes and stateful sets1

Contributing factor Overlapping failures4 Unverified assumption2 Excessive blast radius2 Single point of failure2

Symptom Service outage6 Pods Pending4 Scale-up failure3 High error rate3

Organisation 2i2c1 Devsisters1 GitLab1 Grafana Labs1 loveholidays

Cases

IL-0119 ChatGPT web service pods crash-looped on failing health checks after a Cosmos DB failure, leaving too few healthy pods to serve the web layerOpenAI · 2025
IL-0084 Packet drops on a bare metal provider's vSwitch forced Namespace to rebuild its fra3 region and left global capacity short, and the next day slow NVMe writes on a control plane node drove the Kubernetes API server to very high CPUNamespace · 2024
IL-0145 A power sag and chiller failure in one Australia East availability zone took down the AKS control plane's compute and its non-zone-redundant SQL Database operation queue, blocking AKS create, update and delete operations in Australia East and Australia SoutheastMicrosoft Azure · 2023
IL-0096 A GKE node pool pinned to the europe-west2-b zone could not scale for 6.5 hours because Google ran out of resources in that zone (ZONE_RESOURCE_POOL_EXHAUSTED)2i2c · 2022
IL-0205 The GKE cluster autoscaler could not add nodes to GitLab.com's production cluster because Google Cloud ran out of SSD persistent disks in us-east1GitLab · 2021

View on the graph

Ignored early signal1

Unpatched known bug1

Limited debug access1

Monitoring blind spot1

No capacity planning1

Pages not delivered1

Health check tied to a dependency1

Unrepresentative staging1

Autoscaler misbehaviour1

Control plane lag1

Control plane unresponsive1

Container crash loop1

Deploys failing1

Intermittent connection failures1

Probe failures1

1

Microsoft Azure1

IL-0172 An AWS Tokyo data centre cooling failure took down 6 of 60 CockroachDB nodes on Kubernetes within 22 minutes, and ranges lost quorum because the Helm-deployed cluster had no availability zone locality configuredDevsisters · 2021

IL-0193 A GCP persistent disk degradation took down a Cassandra StatefulSet whose OrderedReady policy blocked recovery, and Metrictank could not reconnect to the recreated pods, causing a 23-hour Graphite query outageGrafana Labs · 2019

IL-0014 A GKE upgrade to a withdrawn version stalled on slow drains and a stuck node pool, then broke an ingress and confused the HPAloveholidays · 2019