Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases51 cases

19 to 23 of 51

  1. IL-0042 Application logs stopped reaching Elasticsearch and OpenSearch for hours and were lost when OpenSearch ran out of disk space
    UK Ministry of Justice2024
    Symptoms

    Kibana stopped showing new application logs at 13:45. At 14:27 a user told the Cloud Platform team in the #ask-cloud-platform channel that Kibana had received no logs since then.

    Monitoring outage
    Trigger

    OpenSearch ran out of disk space and index writes were blocked.

    Root cause
    Capacity exhaustion
    Component
    Telemetry agentElasticsearch or OpenSearch
    Contributing factor
    No alert on limit
    Detection

    A user reported in #ask-cloud-platform at 14:27 that Kibana had received no logs since 13:45. The report records 12:10, when the ClusterIndexWritesBlocked alert first fired, as the first detection.

    Remediation
    • restarted all fluent-bit containers (logs flowed for a few minutes, then the error came back)
    • increased OpenSearch disk space from 8000 to 12000
    • configured fluent-bit to stop sending logs to OpenSearch while the root cause was investigated
    Action items
    • Look into index dating issues in OpenSearch and Elasticsearch
    • Add high-priority alerts for Elasticsearch and OpenSearch
    • Re-introduce OpenSearch into live logging
    • Investigate the fluent-bit "failed to flush chunk" error
    View case
  2. IL-0044 Prometheus OOMKilled in a loop while replaying its write-ahead log after a kube-prometheus-stack upgrade
    Saifeddine Rajhi2024
    Symptoms

    After an upgrade of the kube-prometheus-stack Helm chart, several pods stayed not ready. Among them was prometheus-prometheus-operator-prometheus-0, showing 3/4 Running and a recent termination.

    OOMKilledProbe failuresMonitoring outage
    Trigger

    During the kube-prometheus-stack Helm chart upgrade, Prometheus had to replay its write-ahead log on startup, and the replay needed more memory than its limit allowed.

    Root cause
    Capacity exhaustion
    Component
    PrometheusContainer memory limit
    Contributing factor
    Unknown limitNo headroom
    Detection

    The team noticed pods not ready after the Helm chart upgrade and found the OOMKilled termination with kubectl describe pod.

    Remediation
    • ran promtool tsdb analyze inside the Prometheus pod to find high-cardinality metrics such as haproxy_server_http_responses_total
    Action items
    • Verify that the memory overhead during WAL replay stays within acceptable limits, for example 10 to 15%, and do not run Prometheus close to its memory limit
    • Optimise storage, scraping and remote write configuration and upgrade to newer Prometheus releases
    • Recover automatically from OOM crash loops, for example by deleting the WAL on OOM events
    • Enforce scrape limits so Prometheus stops scraping targets when memory use passes a threshold
    Lessons

    Size Prometheus for its restart, not only for steady state: WAL replay can need far more memory than normal running, and a Prometheus that already runs near its limit can be OOM-killed during the replay. Much of Prometheus memory use comes from cardinality, which promtool tsdb analyze can measure even when Prometheus itself is down.

    View case
  3. IL-0182 Too few CoreDNS replicas on EKS were overloaded during traffic spikes, raising latency and cutting the network for task pods, and the over-provisioned, tightly spread CoreDNS pods added later blocked Karpenter consolidation
    이언철2024
    Symptoms

    When traffic changed sharply, CoreDNS came under heavy load, and the cluster saw network problems and a sudden rise in latency. The horizontal pod autoscaler then added pods, database connections surged, and the impact spread to the whole service.

    High latencyIntermittent connection failuresAutoscaler misbehaviour
    Trigger

    Sharp changes in traffic loaded the small number of CoreDNS pods beyond what they could handle.

    Root cause
    Capacity exhaustion
    Component
    CoreDNSKarpenterHorizontal pod autoscalerPod scheduling and placement
    Contributing factor
    Automation amplificationHarmful manual action
    Remediation
    • Set up the Cluster Proportional Autoscaler for CoreDNS with nodesPerReplica and coresPerReplica chosen for the instance types in use
    • Spread CoreDNS with soft topologySpreadConstraints (ScheduleAnyway) instead of anti-affinity so Karpenter could consolidate nodes
    • Added alert thresholds on CoreDNS error rate, panics, request duration and request rate
    • Deployed NodeLocal DNSCache so each node answers DNS from a local cache and forwards misses to the main CoreDNS
    Lessons

    A fixed, small number of CoreDNS pods cannot follow a cluster whose nodes and pods change quickly with traffic, and the damage spreads: DNS latency triggers autoscaling, which in turn floods the database with connections. Simply adding many CoreDNS replicas created a different problem, because with tainted nodes and a maxSkew: 1 spread constraint the pods piled onto a few nodes and blocked Karpenter consolidation. Scaling CoreDNS in proportion to the cluster and putting a DNS cache on every node handled both.

    View case
  4. IL-0046 Prometheus restarted again and again because its WAL replay took longer than the startup probe allowed
    UK Ministry of Justice2024
    Symptoms

    At 12:32 Prometheus on the Cloud Platform was not available after a planned change, and the Prometheus pod had restarted several times. The platform lost its monitoring and had missing metrics for the period.

    Monitoring outageProbe failures
    Trigger

    A planned change restarted Prometheus at a time when the WAL count was high.

    Root cause
    Capacity exhaustion
    Component
    Prometheus
    Contributing factor
    Unreviewed defaultsUnknown limit
    Detection

    The team saw that the Prometheus pod had restarted several times after the planned change.

    Remediation
    • raised the Prometheus startupProbe threshold to 30 minutes
    Action items
    • Perform planned Prometheus restarts when the WAL count is lower
    • Set the default CPU and memory requests to meet maximum usage
    • Create a test setup that recreates the live WAL count
    • Explore the memory-snapshot-on-shutdown and auto-gomaxprocs feature flags
    • Explore storing WAL files remotely in a different location
    • Look into a blue-green Prometheus to test changes on a live-like setup before applying them to live
    • Spike into Amazon Managed Prometheus
    View case
  5. IL-0110 A Grafana Agent pod's high memory use and crash-looping Hangfire pods drove a Kubernetes node into system OOM and NotReady, twice in one day
    DFDS2024
    Symptoms

    At 10:01 UTC on 2024-03-13 a Kubernetes node entered NotReady state, with CPU requests at 78% and memory requests at 82%. An alert fired in Slack at 10:03 and was picked up at 10:05. An attempt to SSH into the node at 10:13 failed because it was unresponsive; it started responding again at 10:19.

    Node NotReadyContainer crash loopOOMKilledMonitoring outage
    Trigger

    Pods of a dev container that used Hangfire to trigger jobs failed with OOM and crash-looped on a node where a Grafana Agent pod was already using a lot of memory.

    Root cause
    Capacity exhaustion
    Component
    Telemetry agentContainer memory limitApplication database
    Contributing factor
    Excessive blast radiusNo resource requests or limitsHarmful manual actionMonitoring blind spot
    Detection

    A Slack alert fired two minutes after the node went NotReady and was picked up two minutes later; the second node's NotReady at 15:03 also raised a Slack alert.

    Remediation
    • marked the EC2 instance unhealthy so a new node started
    • cordoned and force-drained the NotReady node and terminated its EC2 instance
    • removed the new LimitRange from the Grafana Agent namespace and restarted the Grafana Agent pod
    • scaled the dev container to 0 replicas, after which EC2 network traffic returned to normal
    • after the second NotReady, scaled the dev container to 0 replicas again and added a new node
    • emailed the team that owns the dev container, which confirmed it was working on a fix
    Action items
    • Move grafana-agent to the monitoring node group
    • Add alerts for when bandwidth across the cluster reaches a certain threshold
    • Investigate whether memory limits can be set on the grafana-agent Helm chart
    • Investigate grafana-agent memory usage
    • Ask Grafana support whether grafana-agent should use a PVC
    • Consider pushing node logs to Loki with limited retention
    • Create a wiki page with common metrics and log queries to check when investigating issues with Kubernetes nodes
    Lessons

    Dev workloads mixed with production let one crash-looping dev container take down nodes that production also used, and a monitoring agent outside its dedicated node group added to the memory pressure. Scaling the suspect container back up before its owners had fixed it brought the failure back on the replacement node. The team also noted that action items from earlier post-mortems need follow-up, and that knowledge of useful CloudWatch metrics had not been shared, which made the diagnosis harder.

    View case
Check your cluster for this

These 51 cases involve Container memory limit and Node pool. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion51 cases

Component
Container memory limit12Node pool11Kubernetes API server9CoreDNS8Application database6kubelet6Prometheus6Telemetry agent6Client-side DNS resolver and cache4etcd4Pod scheduling and placement4In-memory cache3Horizontal pod autoscaler3Ingress controller3VPC subnet or pod address range3Cluster autoscaler2conntrack table2Container runtime2Container CPU limit2Karpenter2kube-dns2kube-proxy2Linux kernel2Node root disk2Persistent volumes and stateful sets2Shared network filesystem2AWS VPC CNI2Admission webhook1CI system1Cilium1Cloud controller manager1Credential broker for pods1Elasticsearch or OpenSearch1GKE control plane1HAProxy1Helm1Istio1Jobs and CronJobs1Language runtime settings1NAT gateway1Cluster authentication provider1Container registry and its credentials1Service discovery1Training data loader1VPC DNS resolver1
Contributing factor
Unreviewed defaults17Monitoring blind spot13Unknown limit12Single point of failure12Retry storm10Automation amplification7Excessive blast radius7Harmful manual action7Unverified assumption6No capacity planning6No alert on limit5No resource requests or limits5Ignored early signal3No caching3No headroom3Unrepresentative staging3Limited debug access2No staged rollout2Ineffective rollback2Change during peak hours1No runbook1Overlapping failures1
Organisation
UK Ministry of Justice52i2c3Zalando3Backpackr2loveholidays2Palark2Scaleway2Adevinta1Airbnb1BC Government1Blue Matador1Buildkite1Civis Analytics1Datadog1DFDS1DigitalOcean1Fullstaq1GitLab1Google Cloud1Honeycomb1Hugging Face1Kevin1Lablup Inc1LINE1Linode1Magic Pages1Mercari1Moonlight1Neon1Onkar Raghunath Shelke1PostHog1RebelMouse1Render1Saifeddine Rajhi1Saltside1Temporal1Universe1unknown1이언철1
  1. IL-0071 Worker node running out of inodes while disk space looked fine, because containerd kept a full node_modules tree in each of several image layer snapshots
    Onkar Raghunath Shelke2026
  2. IL-0102 CoreDNS pods on a production EKS cluster ran out of memory under a burst of new pods, endpoints and nodes, taking down the Buildkite web UI, APIs and job dispatch
    Buildkite2026
  3. IL-0155 Some DOKS clusters in NYC1 saw intermittent DNS timeouts and NodeNotReady events because CoreDNS ran on shared-CPU Droplet worker nodes
    DigitalOcean2026
  4. IL-0137 Training initialisation on a 60-node NVIDIA B200 cluster took more than 8 hours instead of minutes because fragmented small random I/O from all nodes saturated the metadata service of the VAST Data NFS storage
    Lablup Inc2026
  5. IL-0181 A traffic spike from an external community post pushed product API pods into CPU throttling under a 1-core limit, and an e-commerce product page slowed down while every pod stayed Running
    Kevin2026
  6. IL-0129 Throughput of the 384-GPU SmolLM3 pretraining run collapsed because shared WekaFS storage evicted dataset shards to S3, then kept dropping on local NVMe because the nanotron dataloader built an index that grew with the step count
    Hugging Face2025
  7. IL-0100 PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queries
    PostHog2025
  8. IL-0048 DNS lookups timed out in a large EKS cluster when CoreDNS hit the 1024 packets-per-second link-local limit, driven by sudo hostname lookups
    unknown2025
  9. IL-0067 Redis filled its persistent volume, corrupted its append-only file and crash-looped on every restart
    BC Government2025
  10. IL-0080 kube-dns on Mercari's GKE clusters could not keep up with peak-hour DNS query rates, and a Sev1 production outage followed when several services failed DNS resolution
    Mercari2025
  11. IL-0082 Losing one CoreDNS pod during a planned EKS node reroll overloaded the remaining CoreDNS pods, causing DNS timeouts and errors across multiple services at RebelMouse
    RebelMouse2025
  12. IL-0006 VPC subnets ran out of IP addresses because the AWS VPC CNI kept IPs on full nodes, and WARM_IP_TARGET changes then blocked pod starts after every pod deletion
    Neon2025
  13. IL-0099 User servers on 2i2c's Earthscope hub took over 15 minutes to start and kernels were killed during a workshop, because a profile option guaranteed 8 GB of memory with no limit on 32 GB nodes
    2i2c2025
  14. IL-0170 During a Lunar New Year event, traffic more than three times normal filled the nf_conntrack table on the EKS node running the Istio ingress gateway, dropping packets and taking down Backpackr's web and app services
    Backpackr2025
  15. IL-0027 All CoreDNS pods landed on one node, the node ran out of memory, and DNS for the whole service went down
    Backpackr2025
  16. IL-0085 A control plane node of Magic Pages' Kubernetes cluster on Hetzner Cloud ran out of memory, the cluster's own calls then used up the Hetzner API rate limit, and all customer websites were down for about 7.5 hours
    Magic Pages2025
  17. IL-0204 An etcd memory spike overloaded the control plane of a Render Frankfurt cluster, and DNS then failed because CoreDNS ran on the rebooted control plane nodes and could not start without the API server
    Render2024
  18. IL-0007 The AWS VPC CNI could not assign IP addresses to pods as an EKS cluster ran out of IPv4 space
    Adevinta2024
  19. IL-0042 Application logs stopped reaching Elasticsearch and OpenSearch for hours and were lost when OpenSearch ran out of disk space
    UK Ministry of Justice2024
  20. IL-0044 Prometheus OOMKilled in a loop while replaying its write-ahead log after a kube-prometheus-stack upgrade
    Saifeddine Rajhi2024
  21. IL-0182 Too few CoreDNS replicas on EKS were overloaded during traffic spikes, raising latency and cutting the network for task pods, and the over-provisioned, tightly spread CoreDNS pods added later blocked Karpenter consolidation
    이언철2024
  22. IL-0046 Prometheus restarted again and again because its WAL replay took longer than the startup probe allowed
    UK Ministry of Justice2024
  23. IL-0110 A Grafana Agent pod's high memory use and crash-looping Hangfire pods drove a Kubernetes node into system OOM and NotReady, twice in one day
    DFDS2024
  24. IL-0168 Simultaneous cluster auto-upgrades overloaded Scaleway's Kubernetes API, and a node authentication retry loop kept Kapsule and Kosmos nodes NotReady in FR-PAR
    Scaleway2024
  25. IL-0169 The Scaleway Kubernetes API in FR-PAR ran out of memory under simultaneous deployments, and because it also handled node authentication, Kapsule and Kosmos nodes went NotReady
    Scaleway2024
  26. IL-0202 A burst of queries and database writes exhausted connection pools and memory on Honeycomb's ingest hosts, and Kubernetes crash-loop back-off kept failed hosts offline so ingest capacity stayed too low to recover
    Honeycomb2023
  27. IL-0043 Application logs missing in Kibana after a newer Fluent Bit version changed its chunk drop strategy
    UK Ministry of Justice2023
  28. IL-0016 Prometheus on the live cluster ran out of memory and was OOMKilled, and the monitoring node it ran on went NotReady
    UK Ministry of Justice2023
  29. IL-0028 Temporal History service latency rose because its pods were CPU-throttled while CPU usage read below 60 percent
    Temporal2023
  30. IL-0195 The kube-apiservers of a cluster with more than 200 nodes ran out of memory and crashed one after another under mass LIST requests from Cilium agents
    Palark2023
  31. IL-0165 Some LKE users in Frankfurt could not reach their kube-apiservers with kubectl after backend services failed under memory constraints
    Linode2023
  32. IL-0194 After a kube-apiserver restart, pod LIST requests from every pod of a Vector DaemonSet nearly tripled etcd memory and restarted all master nodes
    Palark2023
  33. IL-0070 Critical pods on core nodes evicted when Prometheus began scraping about 200 nodes after a dask-gateway scale-up
    2i2c2023
  34. IL-0140 The GKE API in us-central1 returned errors and high latency for 3 hours 19 minutes after an unexpected rise in backend traffic that client retries made worse
    Google Cloud2022
  35. IL-0206 Consul agents in GitLab.com's GKE pods were repeatedly OOM-killed, so the Rails pods lost database replica discovery and sent every query to the primary
    GitLab2022
  36. IL-0094 Student pods filled the shared nodes of a JupyterHub cluster and starved the hub's core pods, freezing the hub
    2i2c2020
  37. IL-0093 Ingress controller pods crash-looped one after another because the leader pod had exhausted its shared memory
    UK Ministry of Justice2020
  38. IL-0176 Prometheus in a 235-node Kubernetes cluster crash-looped because over two million series exhausted host memory and the kernel OOM killer chose it first, as Burstable QoS gave it oom_score_adj 999
    LINE2020
  39. IL-0013 The conntrack table filled up on downsized GKE nodes and a high-traffic aggregation proxy saw climbing connection resets
    loveholidays2020
  40. IL-0015 A GKE cluster's /16 pod address range ran out at 256 nodes, leaving new pods Pending and node scale-ups failing
    loveholidays2020
  41. IL-0033 Uneven load across kube-dns pods and a DNS packets-per-second limit pushed service latency to 20 seconds
    Airbnb2019
  42. IL-0026 Kubernetes nodes hit SystemOOM because fluentd-sumologic log shipper pods had no memory limits
    Blue Matador2019
  43. IL-0002 CI builder pods could not get AWS IAM credentials because kubelet fell minutes behind its API rate limit
    Zalando2019
  44. IL-0076 Kubernetes masters went down after Dex filled etcd with AuthRequest objects and deleting its namespace started an endless list-and-timeout loop
    Fullstaq2019
  45. IL-0037 Image registry rate-limited a whole region after a crash-looping DaemonSet with imagePullPolicy Always pulled through three NAT instances
    Datadog2019
  46. IL-0064 CPU-heavy pods packed onto one Google Cloud node drove it to kernel panics, and the crash moved from node to node until the site was down
    Moonlight2019
  47. IL-0075 Kubernetes API servers pegged on CPU when the cluster grew to about 290 nodes, traced to Datadog agents polling the API
    Civis Analytics2019
  48. IL-0001 Cluster-wide DNS outage after CoreDNS pods were OOM-killed again and again under a burst of DNS queries
    Zalando2019
  49. IL-0031 Kubernetes API server OOM-killed, leaving the Skipper ingress without routes; the talk points to a misconfigured CronJob manifest
    Zalando2018
  50. IL-0036 A reindex Job that kept failing exhausted one node after another until the whole cluster was down
    Universe2017
  51. IL-0040 Parallel Helm installs saturated small gp2 root volumes on a kops cluster and left all three workers NotReady with the kubelet or Docker down
    Saltside2017