Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Component

Container memory limit

컨테이너 메모리 한도 · memory-limit

18 cases · 2019 to 2026

Root cause Capacity exhaustion12 Configuration change3 Software defect2 DNS or network fault1

Contributing factor No resource requests or limits5 Single point of failure5 Excessive blast radius4 Unreviewed defaults4

Symptom OOMKilled13 Service outage8 Monitoring outage6 High error rate4

Organisation 2i2c2 AdGuard VPN1 Backpackr1 Blue Matador1 Buildkite

Cases, by severity

IL-0102 CoreDNS pods on a production EKS cluster ran out of memory under a burst of new pods, endpoints and nodes, taking down the Buildkite web UI, APIs and job dispatchBuildkite · 2026
Root cause Capacity exhaustion
IL-0063 VPN outage after recreating a Redis StatefulSet restarted the whole Redis cluster without readiness checksAdGuard VPN · 2025
Root cause Configuration change
IL-0099 User servers on 2i2c's Earthscope hub took over 15 minutes to start and kernels were killed during a workshop, because a profile option guaranteed 8 GB of memory with no limit on 32 GB nodes

View on the graph

Monitoring blind spot4

Unverified assumption3

No alert on limit3

Ignored early signal2

Harmful manual action2

Automation amplification1

No health probes1

No staged rollout1

Overlapping failures1

Unrepresentative staging1

Probe failures4

DNS resolution failures3

Intermittent timeouts2

Autoscaler misbehaviour1

Control plane unresponsive1

Container crash loop1

Database overload1

Image pull failures1

Intermittent connection failures1

Sustained high node CPU1

Load imbalance1

1

Saifeddine Rajhi1

UK Ministry of Justice1

2i2c · 2025

Root cause Capacity exhaustion

IL-0027 All CoreDNS pods landed on one node, the node ran out of memory, and DNS for the whole service went downBackpackr · 2025

Root cause Capacity exhaustion

IL-0044 Prometheus OOMKilled in a loop while replaying its write-ahead log after a kube-prometheus-stack upgradeSaifeddine Rajhi · 2024

Root cause Capacity exhaustion

IL-0110 A Grafana Agent pod's high memory use and crash-looping Hangfire pods drove a Kubernetes node into system OOM and NotReady, twice in one dayDFDS · 2024

Root cause Capacity exhaustion

IL-0169 The Scaleway Kubernetes API in FR-PAR ran out of memory under simultaneous deployments, and because it also handled node authentication, Kapsule and Kosmos nodes went NotReadyScaleway · 2024

Root cause Capacity exhaustion

IL-0123 A response buffer allocated anew in a loop made OpenAI's routing layer nodes hit their memory limits and fail readiness checks on a morning with more completions than any day before, causing 502 and 503 errors across all modelsOpenAI · 2023

Root cause Software defect

IL-0016 Prometheus on the live cluster ran out of memory and was OOMKilled, and the monitoring node it ran on went NotReadyUK Ministry of Justice · 2023

Root cause Capacity exhaustion

IL-0165 Some LKE users in Frankfurt could not reach their kube-apiservers with kubectl after backend services failed under memory constraintsLinode · 2023

Root cause Capacity exhaustion

IL-0191 A label selector accidentally added to the Loki distributor Service matched only the new pods of a rollout, which took all write traffic, were OOM-killed and stalled the rolloutGrafana Labs · 2022

Root cause Configuration change

IL-0206 Consul agents in GitLab.com's GKE pods were repeatedly OOM-killed, so the Rails pods lost database replica discovery and sent every query to the primaryGitLab · 2022

Root cause Capacity exhaustion

IL-0094 Student pods filled the shared nodes of a JupyterHub cluster and starved the hub's core pods, freezing the hub2i2c · 2020

Root cause Capacity exhaustion

IL-0199 A non-essential container exceeding its memory limit made GitHub's production pods unavailable, and their restarts failed because a DNS maintenance change had left the clusters unable to reach the image registryGitHub · 2020

Root cause DNS or network fault

IL-0196 Adding a second Redis replica and lowering memory limits in one RedisFailover change restarted the only Redis pod that held data, and Sentinel promoted the empty new replica, erasing the dataPalark · 2020

Root cause Software defect

IL-0026 Kubernetes nodes hit SystemOOM because fluentd-sumologic log shipper pods had no memory limitsBlue Matador · 2019

Root cause Capacity exhaustion

IL-0035 Worker nodes went NotReady after a new ElastAlert rule without memory limits used up node memoryNU.nl · 2019

Root cause Configuration change

IL-0001 Cluster-wide DNS outage after CoreDNS pods were OOM-killed again and again under a burst of DNS queriesZalando · 2019

Root cause Capacity exhaustion