Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases51 cases

24 to 28 of 51

  1. IL-0168 Simultaneous cluster auto-upgrades overloaded Scaleway's Kubernetes API, and a node authentication retry loop kept Kapsule and Kosmos nodes NotReady in FR-PAR
    Scaleway2024
    Symptoms

    At 01:04 CET on 13 February 2024, the Kubernetes API behind Scaleway's managed Kubernetes services Kapsule and Kosmos in the Paris region (FR-PAR) spiked in RAM and CPU use and ran out of memory. Nodes in FR-PAR could no longer authenticate to their control plane, and some of them went "Not Ready". Customers on both shared (mutualized) and dedicated environments were affected.

    Node NotReadyControl plane lagOOMKilled
    Trigger

    A new Kubernetes release was made available to all users on 12 February 2024, and during the night of 12 to 13 February many clusters triggered their auto-upgrade at the same time.

    Root cause
    Capacity exhaustion
    Component
    Kubernetes API server
    Contributing factor
    Retry stormAutomation amplificationNo staged rollout
    Detection

    The timeline's first entry, at 01:04 CET, records the Kubernetes API's RAM and CPU spikes and out-of-memory state, with nodes unable to authenticate to their control plane and some becoming "Not Ready". The first recorded action, 42 minutes later at 01:46, was to increase the resources allocated to the API.

    Remediation
    • increased the resources allocated to the Kubernetes API at 01:46 and again at 02:20
    • applied API traffic restrictions at 02:30 to smooth the incoming traffic
    • removed the traffic restrictions progressively in 50-100 batches from 03:20
    • ran a script at 03:40 to replace the failing nodes created during the congestion
    • rolled back to the former node authentication mechanism, first on a subset of clusters from 15:45 and on all clusters by 19:41 on 19 February
    • disabled cluster upgrade and configuration update features until 10:30 on 20 February
    Action items
    • Increase the resources allocated to the Kubernetes API
    • Roll back to the former node authentication mechanism
    • Implement a local authentication cache on nodes
    • Rework the resource allocation strategy to behave better under load spikes
    • Add new request queuing mechanisms for Kubernetes
    • Separate the different components to reduce side effects under high load or failure
    • Monitor selected metrics closely to identify anomalous patterns earlier
    • Refund customers who deployed nodes during the incident but could not use them normally
    Lessons

    Letting many clusters auto-upgrade at the same moment after a release turns routine upgrades into one coordinated load spike on the shared Kubernetes API. A node authentication mechanism that retries without letting up turns a slow API into an unavailable one, and auto-healing that replaces NotReady nodes amplifies the problem. The congestion pattern had already appeared three weeks earlier, and short-term mitigations did not stop it from recurring, so Scaleway rolled back the new authentication mechanism. Scaleway notes that as the number of customer clusters and nodes grows sharply, traffic control mechanisms have to improve to prevent congestion.

    View case
  2. IL-0169 The Scaleway Kubernetes API in FR-PAR ran out of memory under simultaneous deployments, and because it also handled node authentication, Kapsule and Kosmos nodes went NotReady
    Scaleway2024
    Symptoms

    At 23:48 CET on 24 January 2024 the Scaleway Kubernetes API for the FR-PAR region ran out of memory. At the same minute Scaleway saw requests to its API Gateway spike.

    Node NotReadyOOMKilledIntermittent timeoutsMonitoring outage
    Trigger

    Simultaneous deployments in the FR-PAR region pushed the Scaleway Kubernetes API out of memory.

    Root cause
    Capacity exhaustion
    Component
    Container memory limitkubelet
    Contributing factor
    Single point of failureRetry storm
    Detection

    Increased load on the API Gateway was detected at 23:48, the minute the incident began, and the incident was escalated to on-call engineers at 00:00:10.

    Remediation
    • raised the memory limit and garbage collection thresholds of the Kubernetes managed services
    • deployed additional replicas of key services
    • purged the registry cache and monitored the clusters as they stabilised
    • adjusted service configurations after the incident to keep the platform stable
    Action items
    • Implement a local authentication cache
    • Further develop node authentication
    • Enhance monitoring, adjust rate limiting strategies and improve incident response protocols
    Lessons

    Because the Kubernetes API also handled node authentication, its memory exhaustion made some nodes NotReady, and the endless retries of those nodes added load on the API Gateway. Scaleway plans a local authentication cache and further work on node authentication.

    View case
  3. IL-0202 A burst of queries and database writes exhausted connection pools and memory on Honeycomb's ingest hosts, and Kubernetes crash-loop back-off kept failed hosts offline so ingest capacity stayed too low to recover
    Honeycomb2023sev1
    Symptoms

    Standard alerting paged Honeycomb's on-call engineers about the performance and stability of event ingestion, and the automated load-shedding mechanisms tripped and sent further notifications. The status page described a higher than normal error rate and response time on the ingestion service.

    High error rateHigh latencyContainer crash loopService outageDatabase overload
    Trigger

    An abnormally large number of queries arrived from many separate connections within a few minutes.

    Root cause
    Capacity exhaustion
    Component
    Application databaseIn-memory cache
    Contributing factor
    Retry stormAutomation amplification
    Detection

    Standard alerting notified the on-call engineers about ingestion performance and stability, and the automated load-shedding mechanisms added notifications when they tripped. The first status post, about alerts on a higher than normal error rate and response time, went up at 18:28 PST (02:28 UTC), 13 minutes after the 02:15 UTC start.

    Remediation
    • Scaled the ingest fleet up manually and aggressively
    • Cut off most ingest for a few minutes so the ingest hosts could come back with enough capacity
    • Temporarily lowered the rate limits for the most aggressive traffic sources until their traffic subsided
    Action items
    • Keep the ingest fleet scaled up to prevent a repeat while longer-term fixes are worked on
    • Decide on the projects that will address these failure paths
    Lessons

    Load shedding and auto-scaling were in place, yet a burst that arrived faster than the cache could spread its writes still took the ingest hosts down. During recovery, Kubernetes' crash loop back-off worked against the operators: the hosts that had failed stayed offline while the fleet needed every host. Clients that retry whole OTel batches after a partial drop can multiply traffic many times over, so per-source rate limits were what finally stabilised ingest.

    View case
  4. IL-0043 Application logs missing in Kibana after a newer Fluent Bit version changed its chunk drop strategy
    UK Ministry of Justice2023
    Symptoms

    At 09:14 on 4 August 2023, users of the Cloud Platform reported in the #ask-cloud-platform Slack channel that Kibana was missing their application logs for long periods.

    Monitoring outage
    Trigger

    The newer Fluent Bit version in use had changed its chunk drop strategy; the fix that followed handled memory buffer overflow.

    Root cause
    Capacity exhaustion
    Component
    Telemetry agent
    Contributing factor
    No alert on limitMonitoring blind spot
    Detection

    Users reported long periods of missing logs in Kibana in the #ask-cloud-platform Slack channel at 09:14; the incident was declared at 10:09.

    Remediation
    • configured Fluent Bit to handle memory buffer overflow by writing buffered logs to the file system
    • made Fluent Bit flush logs in smaller chunks
    Action items
    • Push notifications from the logging clusters to #lower-priority-alerts (#4704)
    • Add an integration test that checks logs are being sent to the logging cluster
    View case
  5. IL-0016 Prometheus on the live cluster ran out of memory and was OOMKilled, and the monitoring node it ran on went NotReady
    UK Ministry of Justice2023
    Symptoms

    At 14:05 on 25 July 2023, a high-priority PagerDuty alert from Pingdom reported that the "Prometheus - live healthcheck" was DOWN. Prometheus on the live cluster was unavailable, and the Cloud Platform lost its monitoring for a while.

    Monitoring outageOOMKilledNode NotReady
    Trigger

    The growing number of namespaces and Prometheus rules raised Prometheus' memory needs beyond what its node instance size could hold.

    Root cause
    Capacity exhaustion
    Component
    PrometheusNode poolContainer memory limit
    Contributing factor
    No alert on limit
    Detection

    A Pingdom health check on Prometheus failed and raised a high-priority PagerDuty alert.

    Remediation
    • drained the monitoring node, moving Prometheus to another monitoring node
    • changed the node instance size to r6i.4xlarge
    • raised the Prometheus memory limit to 100Gi
    • set the Prometheus container limits to 12 CPU cores and 110Gi of memory
    Action items
    • Add an alert on node memory usage and on a pod using up most of a node's memory (issue #4538)
    View case
Check your cluster for this

These 51 cases involve Container memory limit and Node pool. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion51 cases

Component
Container memory limit12Node pool11Kubernetes API server9CoreDNS8Application database6kubelet6Prometheus6Telemetry agent6Client-side DNS resolver and cache4etcd4Pod scheduling and placement4In-memory cache3Horizontal pod autoscaler3Ingress controller3VPC subnet or pod address range3Cluster autoscaler2conntrack table2Container runtime2Container CPU limit2Karpenter2kube-dns2kube-proxy2Linux kernel2Node root disk2Persistent volumes and stateful sets2Shared network filesystem2AWS VPC CNI2Admission webhook1CI system1Cilium1Cloud controller manager1Credential broker for pods1Elasticsearch or OpenSearch1GKE control plane1HAProxy1Helm1Istio1Jobs and CronJobs1Language runtime settings1NAT gateway1Cluster authentication provider1Container registry and its credentials1Service discovery1Training data loader1VPC DNS resolver1
Contributing factor
Unreviewed defaults17Monitoring blind spot13Unknown limit12Single point of failure12Retry storm10Automation amplification7Excessive blast radius7Harmful manual action7Unverified assumption6No capacity planning6No alert on limit5No resource requests or limits5Ignored early signal3No caching3No headroom3Unrepresentative staging3Limited debug access2No staged rollout2Ineffective rollback2Change during peak hours1No runbook1Overlapping failures1
Organisation
UK Ministry of Justice52i2c3Zalando3Backpackr2loveholidays2Palark2Scaleway2Adevinta1Airbnb1BC Government1Blue Matador1Buildkite1Civis Analytics1Datadog1DFDS1DigitalOcean1Fullstaq1GitLab1Google Cloud1Honeycomb1Hugging Face1Kevin1Lablup Inc1LINE1Linode1Magic Pages1Mercari1Moonlight1Neon1Onkar Raghunath Shelke1PostHog1RebelMouse1Render1Saifeddine Rajhi1Saltside1Temporal1Universe1unknown1이언철1
  1. IL-0071 Worker node running out of inodes while disk space looked fine, because containerd kept a full node_modules tree in each of several image layer snapshots
    Onkar Raghunath Shelke2026
  2. IL-0102 CoreDNS pods on a production EKS cluster ran out of memory under a burst of new pods, endpoints and nodes, taking down the Buildkite web UI, APIs and job dispatch
    Buildkite2026
  3. IL-0155 Some DOKS clusters in NYC1 saw intermittent DNS timeouts and NodeNotReady events because CoreDNS ran on shared-CPU Droplet worker nodes
    DigitalOcean2026
  4. IL-0137 Training initialisation on a 60-node NVIDIA B200 cluster took more than 8 hours instead of minutes because fragmented small random I/O from all nodes saturated the metadata service of the VAST Data NFS storage
    Lablup Inc2026
  5. IL-0181 A traffic spike from an external community post pushed product API pods into CPU throttling under a 1-core limit, and an e-commerce product page slowed down while every pod stayed Running
    Kevin2026
  6. IL-0129 Throughput of the 384-GPU SmolLM3 pretraining run collapsed because shared WekaFS storage evicted dataset shards to S3, then kept dropping on local NVMe because the nanotron dataloader built an index that grew with the step count
    Hugging Face2025
  7. IL-0100 PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queries
    PostHog2025
  8. IL-0048 DNS lookups timed out in a large EKS cluster when CoreDNS hit the 1024 packets-per-second link-local limit, driven by sudo hostname lookups
    unknown2025
  9. IL-0067 Redis filled its persistent volume, corrupted its append-only file and crash-looped on every restart
    BC Government2025
  10. IL-0080 kube-dns on Mercari's GKE clusters could not keep up with peak-hour DNS query rates, and a Sev1 production outage followed when several services failed DNS resolution
    Mercari2025
  11. IL-0082 Losing one CoreDNS pod during a planned EKS node reroll overloaded the remaining CoreDNS pods, causing DNS timeouts and errors across multiple services at RebelMouse
    RebelMouse2025
  12. IL-0006 VPC subnets ran out of IP addresses because the AWS VPC CNI kept IPs on full nodes, and WARM_IP_TARGET changes then blocked pod starts after every pod deletion
    Neon2025
  13. IL-0099 User servers on 2i2c's Earthscope hub took over 15 minutes to start and kernels were killed during a workshop, because a profile option guaranteed 8 GB of memory with no limit on 32 GB nodes
    2i2c2025
  14. IL-0170 During a Lunar New Year event, traffic more than three times normal filled the nf_conntrack table on the EKS node running the Istio ingress gateway, dropping packets and taking down Backpackr's web and app services
    Backpackr2025
  15. IL-0027 All CoreDNS pods landed on one node, the node ran out of memory, and DNS for the whole service went down
    Backpackr2025
  16. IL-0085 A control plane node of Magic Pages' Kubernetes cluster on Hetzner Cloud ran out of memory, the cluster's own calls then used up the Hetzner API rate limit, and all customer websites were down for about 7.5 hours
    Magic Pages2025
  17. IL-0204 An etcd memory spike overloaded the control plane of a Render Frankfurt cluster, and DNS then failed because CoreDNS ran on the rebooted control plane nodes and could not start without the API server
    Render2024
  18. IL-0007 The AWS VPC CNI could not assign IP addresses to pods as an EKS cluster ran out of IPv4 space
    Adevinta2024
  19. IL-0042 Application logs stopped reaching Elasticsearch and OpenSearch for hours and were lost when OpenSearch ran out of disk space
    UK Ministry of Justice2024
  20. IL-0044 Prometheus OOMKilled in a loop while replaying its write-ahead log after a kube-prometheus-stack upgrade
    Saifeddine Rajhi2024
  21. IL-0182 Too few CoreDNS replicas on EKS were overloaded during traffic spikes, raising latency and cutting the network for task pods, and the over-provisioned, tightly spread CoreDNS pods added later blocked Karpenter consolidation
    이언철2024
  22. IL-0046 Prometheus restarted again and again because its WAL replay took longer than the startup probe allowed
    UK Ministry of Justice2024
  23. IL-0110 A Grafana Agent pod's high memory use and crash-looping Hangfire pods drove a Kubernetes node into system OOM and NotReady, twice in one day
    DFDS2024
  24. IL-0168 Simultaneous cluster auto-upgrades overloaded Scaleway's Kubernetes API, and a node authentication retry loop kept Kapsule and Kosmos nodes NotReady in FR-PAR
    Scaleway2024
  25. IL-0169 The Scaleway Kubernetes API in FR-PAR ran out of memory under simultaneous deployments, and because it also handled node authentication, Kapsule and Kosmos nodes went NotReady
    Scaleway2024
  26. IL-0202 A burst of queries and database writes exhausted connection pools and memory on Honeycomb's ingest hosts, and Kubernetes crash-loop back-off kept failed hosts offline so ingest capacity stayed too low to recover
    Honeycomb2023
  27. IL-0043 Application logs missing in Kibana after a newer Fluent Bit version changed its chunk drop strategy
    UK Ministry of Justice2023
  28. IL-0016 Prometheus on the live cluster ran out of memory and was OOMKilled, and the monitoring node it ran on went NotReady
    UK Ministry of Justice2023
  29. IL-0028 Temporal History service latency rose because its pods were CPU-throttled while CPU usage read below 60 percent
    Temporal2023
  30. IL-0195 The kube-apiservers of a cluster with more than 200 nodes ran out of memory and crashed one after another under mass LIST requests from Cilium agents
    Palark2023
  31. IL-0165 Some LKE users in Frankfurt could not reach their kube-apiservers with kubectl after backend services failed under memory constraints
    Linode2023
  32. IL-0194 After a kube-apiserver restart, pod LIST requests from every pod of a Vector DaemonSet nearly tripled etcd memory and restarted all master nodes
    Palark2023
  33. IL-0070 Critical pods on core nodes evicted when Prometheus began scraping about 200 nodes after a dask-gateway scale-up
    2i2c2023
  34. IL-0140 The GKE API in us-central1 returned errors and high latency for 3 hours 19 minutes after an unexpected rise in backend traffic that client retries made worse
    Google Cloud2022
  35. IL-0206 Consul agents in GitLab.com's GKE pods were repeatedly OOM-killed, so the Rails pods lost database replica discovery and sent every query to the primary
    GitLab2022
  36. IL-0094 Student pods filled the shared nodes of a JupyterHub cluster and starved the hub's core pods, freezing the hub
    2i2c2020
  37. IL-0093 Ingress controller pods crash-looped one after another because the leader pod had exhausted its shared memory
    UK Ministry of Justice2020
  38. IL-0176 Prometheus in a 235-node Kubernetes cluster crash-looped because over two million series exhausted host memory and the kernel OOM killer chose it first, as Burstable QoS gave it oom_score_adj 999
    LINE2020
  39. IL-0013 The conntrack table filled up on downsized GKE nodes and a high-traffic aggregation proxy saw climbing connection resets
    loveholidays2020
  40. IL-0015 A GKE cluster's /16 pod address range ran out at 256 nodes, leaving new pods Pending and node scale-ups failing
    loveholidays2020
  41. IL-0033 Uneven load across kube-dns pods and a DNS packets-per-second limit pushed service latency to 20 seconds
    Airbnb2019
  42. IL-0026 Kubernetes nodes hit SystemOOM because fluentd-sumologic log shipper pods had no memory limits
    Blue Matador2019
  43. IL-0002 CI builder pods could not get AWS IAM credentials because kubelet fell minutes behind its API rate limit
    Zalando2019
  44. IL-0076 Kubernetes masters went down after Dex filled etcd with AuthRequest objects and deleting its namespace started an endless list-and-timeout loop
    Fullstaq2019
  45. IL-0037 Image registry rate-limited a whole region after a crash-looping DaemonSet with imagePullPolicy Always pulled through three NAT instances
    Datadog2019
  46. IL-0064 CPU-heavy pods packed onto one Google Cloud node drove it to kernel panics, and the crash moved from node to node until the site was down
    Moonlight2019
  47. IL-0075 Kubernetes API servers pegged on CPU when the cluster grew to about 290 nodes, traced to Datadog agents polling the API
    Civis Analytics2019
  48. IL-0001 Cluster-wide DNS outage after CoreDNS pods were OOM-killed again and again under a burst of DNS queries
    Zalando2019
  49. IL-0031 Kubernetes API server OOM-killed, leaving the Skipper ingress without routes; the talk points to a misconfigured CronJob manifest
    Zalando2018
  50. IL-0036 A reindex Job that kept failing exhausted one node after another until the whole cluster was down
    Universe2017
  51. IL-0040 Parallel Helm installs saturated small gp2 root volumes on a kops cluster and left all three workers NotReady with the kubelet or Docker down
    Saltside2017