Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Contributing factor

No staged rollout

단계적 배포 부재 · no-staged-rollout

13 cases · 2020 to 2025

Root cause Breaking change in an upgrade5 Configuration change4 Capacity exhaustion2 Software defect2

Component Node operating system image5 Application database3 Cloud load balancer2 Configuration templating2

Symptom Service outage6 High error rate5 Node NotReady3 Container crash loop2

Organisation OpenAI4 Altinity1 Datadog1 DFDS1 Grafana Labs1

Cases

IL-0100 PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queriesPostHog · 2025
Root cause Capacity exhaustion
IL-0122 An invalid value in a configuration shared by several ChatGPT services put pods into crash loops and raised error ratesOpenAI · 2025
Root cause Configuration change
A daily scheduled OS update restarted systemd-networkd on cloud-hosted GPU nodes, which conflicted with OpenAI's networking agent and removed all routes, cutting capacity for ChatGPT and the API

View on the graph

Client-side DNS resolver and cache2

Kubernetes API server2

systemd-networkd2

In-memory cache1

Ingress controller1

Node launch template1

Container memory limit1

Persistent volumes and stateful sets1

Pod scheduling and placement1

Container registry and its credentials1

Telemetry agent1

Database overload2

DNS resolution failures2

Resources deleted2

Control plane lag1

Control plane unresponsive1

GPU unavailable1

Image pull failures1

Intermittent connection failures1

Intermittent timeouts1

Monitoring outage1

Sustained high node CPU1

Throughput degradation1

Microsoft Azure1

IL-0120

OpenAI · 2025

Root cause Breaking change in an upgrade

IL-0079 A GKE node auto-upgrade to an ubuntu_containerd image with a kernel regression left kube-proxy unable to apply its ip6tables rules, so load balancer health checks failed and several clusters lost all ingress trafficAltinity · 2025

Root cause Breaking change in an upgrade

IL-0192 A Crossplane composition merge-policy change for a TLS policy update switched Kubernetes Services from LoadBalancer to ClusterIP and deleted the load balancers behind about 25% of Grafana Cloud servicesGrafana Labs · 2025

Root cause Configuration change

IL-0005 A new telemetry service overloaded the Kubernetes API servers of large clusters and broke DNS-based service discovery for all OpenAI servicesOpenAI · 2024

Root cause Configuration change

IL-0117 A scheduled maintenance and an ingress upgrade on OpenAI's user-facing clusters caused a networking control plane regression that briefly broke the data plane, and the services that restarted then overwhelmed a backend persistence store with a heavy first-start queryOpenAI · 2024

Root cause Breaking change in an upgrade

IL-0168 Simultaneous cluster auto-upgrades overloaded Scaleway's Kubernetes API, and a node authentication retry loop kept Kapsule and Kosmos nodes NotReady in FR-PARScaleway · 2024

Root cause Capacity exhaustion

IL-0198 An automatic Ubuntu security update restarted systemd-networkd, which flushed the IP rules Cilium relied on and cut more than 60 percent of Datadog's Kubernetes nodes off the network across regions and cloud providersDatadog · 2023

Root cause Breaking change in an upgrade

IL-0112 The daily node rollover put DFDS's EKS nodes on an AMI later recalled for a kernel issue, causing "PLEG is not healthy" NodeNotReady flaps and pod network timeoutsDFDS · 2022

Root cause Software defect

IL-0144 A systemd security update installed by unattended-upgrades lost the DNS configuration of Ubuntu 18.04 VMs, so AKS kubelets could not resolve registry names and pods failed with ImagePullBackOffMicrosoft Azure · 2022

Root cause Breaking change in an upgrade

IL-0010 A templating change that dropped its {{ }} delimiters made Argo CD delete all 478 Skyscanner services in every regionSkyscanner · 2021

Root cause Configuration change

IL-0196 Adding a second Redis replica and lowering memory limits in one RedisFailover change restarted the only Redis pod that held data, and Sentinel promoted the empty new replica, erasing the dataPalark · 2020

Root cause Software defect