Incident Library
Applied filters Root causeCapacity exhaustionDomainKubernetesClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases69 cases

44 to 48 of 69

  1. IL-0070 Critical pods on core nodes evicted when Prometheus began scraping about 200 nodes after a dask-gateway scale-up
    2i2c2023
    Symptoms

    The hub pod of the prod JupyterHub on the leap cluster (leap.2i2c.org) restarted unexpectedly during a large node scale-up, and a user opened a support ticket. The ingress-nginx and proxy pods kept running.

    Pod evictionPods Pending
    Trigger

    A user's dask-gateway workload added about 100 dask-worker nodes within a minute, and Prometheus started scraping the metrics exporters on about 200 nodes.

    Root cause
    Capacity exhaustion
    Component
    PrometheusNode pool
    Contributing factor
    No headroomExcessive blast radiusMonitoring blind spot
    Detection

    A user support ticket (2i2c Freshdesk ticket 414) about the leap hub; the engineer then looked at Grafana and the cluster's nodes and events.

    Action items
    • Configure CPU and memory requests for the hub and proxy pods (#2127; at least 100m CPU was proposed)
    • Make the Grafana OOMKiller dashboard work (#2213)
    • Move LEAP cloud infrastructure to shared nodes (#2209)
    • Update resource requests and limits, mainly for ingress-nginx and Prometheus (#2324)
    • Decide whether Prometheus should scrape dask worker pods (#2366)
    Lessons

    Prometheus load grows with the number of nodes, so a burst of autoscaled nodes can push a nearly full core node over the edge and evict the pods that matter most. Core pods that request 10m of CPU get a tenth of the CPU share of busy neighbours that request 100m. Kubernetes events expire after an hour, so evidence of OOM kills needs a dashboard that keeps it.

    View case
  2. IL-0500 UToronto JupyterHub logins and server starts returned 500 errors when the hub's sqlite database disk filled up
    2i2c2022
    Symptoms

    On 31 October 2022 at 6:27 PM Pacific time, users of the UToronto JupyterHub run by 2i2c could no longer log in or start their servers, and the hub answered with 500 errors. Users who were already logged in could keep using the hub.

    High error ratePersistent volume full
    Trigger

    The JupyterHub log file, written to the same 10G disk as the hub's sqlite database, grew until the disk was completely full.

    Root cause
    Capacity exhaustion
    Component
    DatabasePersistent volumes and stateful setsJupyterHub
    Contributing factor
    No alert on limit
    Detection

    A user emailed 2i2c support at 7:13 PM that users could not log in; the team acknowledged the issue at 8:33 PM. There was no alert for the problem.

    Remediation
    • Compressed the hub log file into /tmp from a shell in the hub pod, because it could not be compressed in place on the full disk
    • Copied the compressed log to a local machine so the hub pod could be restarted without losing it
    • Resized the hub database disk (the hub-db-dir PVC) from 10G to 60G
    • Restarted the hub pod, which came back with the larger disk and restored service
    Action items
    • Stop keeping hub logs on the same disk as the hub database
    • Add a Grafana graph of free space on the JupyterHub database disk
    • Alert on elevated error rates
    Lessons

    Kubernetes dynamic volume resizing let the hub database disk grow from 10G to 60G without much effort. The risk of keeping logs on the database disk had been identified during the earlier migration, but the team had no solid process for following it through, and because no alert existed the team had to wait for a person to report the outage.

    View case
  3. IL-0140 The GKE API in us-central1 returned errors and high latency for 3 hours 19 minutes after an unexpected rise in backend traffic that client retries made worse
    Google Cloud2022
    Symptoms

    From 03:03 US/Pacific on 14 June 2022, calls to the GKE API in us-central1 through gcloud and the Google Cloud Console were slow and returned errors. Retrying sometimes succeeded.

    High error rateHigh latencyControl plane lag
    Trigger

    An unexpected increase in backend traffic to the GKE API in us-central1.

    Root cause
    Capacity exhaustion
    Component
    GKE control plane
    Contributing factor
    Retry storm
    Remediation
    • scaled up the GKE control plane to handle the increased traffic
    Lessons

    Client retries against an overloaded management API add to the overload. When the cloud provider's control plane is degraded, running workloads may be fine while cluster operations fail, and the status page offered no workaround beyond advising customers to use another region if they could.

    View case
  4. IL-0206 Consul agents in GitLab.com's GKE pods were repeatedly OOM-killed, so the Rails pods lost database replica discovery and sent every query to the primary
    GitLab2022
    Symptoms

    On 2022-04-25 the Sidekiq Apdex (the share of background jobs that finish within a target time) dropped, and an engineer declared an incident in Slack at 15:13 UTC. Users saw intermittent slowness across GitLab.com, and asynchronous work such as pipeline creation and import and export took unusually long.

    High latencyDatabase overloadOOMKilledHigh error rate
    Trigger

    The Consul agent in GitLab.com's GKE pods had memory demand spikes that did not fit in the pods' 500MB memory limit, and the agent was killed again and again.

    Root cause
    Capacity exhaustion
    Component
    Container memory limitService discoveryDatabase
    Contributing factor
    –
    Detection

    An engineer declared the incident in Slack at 15:13 UTC after the Sidekiq Apdex dropped; the second event was found when monitoring detected an elevated error rate at 01:21 UTC on 2022-04-26.

    Remediation
    • planned at 16:13 to raise the per-user connection limit for the gitlab database user and the pgbouncer backend pools for the main database; changes were applied at 17:26 and confirmed at 17:47
    • raised the pod memory limits from 500MB to 1GB so the Consul agents survive memory spikes
    Action items
    • Monitor the Consul agents on GKE pods to see whether the 1GB memory limit keeps them healthy
    • Investigate the Consul memory demand spikes and what can be done beyond raising the hard memory limit
    Lessons

    When the local Consul agent that Rails uses for database service discovery keeps dying, Rails does not fail; it sends every query to the last known primary. What engineers saw was database saturation and slow background jobs, which pointed the investigation first at a recent pgbouncer pool change. The memory limit of the Consul agent in every pod therefore decided the load on the primary database.

    View case
  5. IL-0445 Chatwork's service on EKS became unstable as conntrack tables on some nodes overflowed and connections were dropped
    Chatwork2021
    Symptoms

    In 2020 Chatwork moved its legacy system from EC2 to EKS, running both sides in parallel with Route53 weighted records. Once more than a certain share of requests was sent to the EKS side, the service suddenly became unstable. On some nodes the conntrack table had gone past its limit and connections to the application were being dropped.

    Intermittent connection failuresDNS resolution failuresIntermittent timeoutsLoad imbalance
    Trigger

    During the migration from EC2 to EKS, Chatwork sent more than a certain share of requests to the EKS side. Later, applying node-local-dns with force_tcp in its configuration made DNS unstable again.

    Root cause
    Capacity exhaustion
    Component
    conntrack tableCoreDNSkube-proxyClient-side DNS resolver and cacheVPC DNS resolver
    Contributing factor
    Unreviewed defaultsUnrepresentative stagingUnknown limit
    Remediation
    • Raised the conntrack maxPerCore value in the kube-proxy configuration
    • Installed dns-autoscaler so CoreDNS ran with more pods
    • Installed node-local-dns as a DaemonSet on every node
    • Replaced force_tcp with prefer_udp in the node-local-dns configuration
    Lessons

    On EKS, dns-autoscaler and node-local-dns made DNS stable, but node-local-dns must not use force_tcp there. Stable DNS also matters for sizing the number of pods correctly, since blocked DNS queries had made the pod count grow.

    View case
Check your cluster for this

These 69 cases involve Container memory limit and Node pool. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion69 cases

Component
Container memory limit15Node pool13CoreDNS10Database10Kubernetes API server10Prometheus8Client-side DNS resolver and cache7kubelet7Telemetry agent6In-memory cache5etcd5Pod scheduling and placement5Horizontal pod autoscaler4Ingress controller4Linux kernel4Persistent volumes and stateful sets4Cloud block storage3CI system3conntrack table3Container CPU limit3Helm3Jobs and CronJobs3kube-proxy3Node root disk3VPC subnet or pod address range3Cluster autoscaler2CNI plugin2Container runtime2JupyterHub2Karpenter2kube-dns2Managed Kubernetes control plane and provider API2Container registry and its credentials2AWS VPC CNI2VPC DNS resolver2Admission webhook1BGP routing1Calico1Cilium1Cloud controller manager1Credential broker for pods1Elasticsearch or OpenSearch1GKE control plane1HAProxy1Liveness and readiness probes1Model inference server1InfiniBand fabric1Istio1Language runtime settings1Model hub1NAT gateway1Network control plane (SDN)1NIC, cable and switch port link1Cluster authentication provider1Pod termination1Service discovery1
Contributing factor
Unreviewed defaults23Monitoring blind spot16Unknown limit15Single point of failure13Excessive blast radius10Harmful manual action10Retry storm10No alert on limit9Unverified assumption8Automation amplification8No resource requests or limits8No capacity planning6No headroom5Ignored early signal4Unrepresentative staging4No caching3Ineffective rollback3Change during peak hours2Change with a delayed effect2Limited debug access2No runbook2No staged rollout2Overlapping failures1Partial rollout1Health check tied to a dependency1
Organisation
UK Ministry of Justice52i2c4Buildkite3IONOS Cloud3Zalando3Backpackr2loveholidays2Palark2Scaleway2Adevinta1Airbnb1BC Government1Binder Project1Blue Matador1Chatwork1Civis Analytics1CyberAgent1Cybozu1Datadog1DFDS1DigitalOcean1freedesktop.org1Fullstaq1GitHub1GitLab1Google Cloud1Honeycomb1incident.io1Kevin1Kubernetes project1LINE1Linode1Magic Pages1Mercari1Microsoft1MonotaRO1Moonlight1MUSINSA1Neon1Onkar Raghunath Shelke1PostHog1RebelMouse1Render1Saifeddine Rajhi1Saltside1Temporal1Tinder1UC Berkeley1Universe1unknown1Wikimedia Foundation1이언철1
  1. IL-0071 Worker node running out of inodes while disk space looked fine, because containerd kept a full node_modules tree in each of several image layer snapshots
    Onkar Raghunath Shelke2026
  2. IL-0102 CoreDNS pods on a production EKS cluster ran out of memory under a burst of new pods, endpoints and nodes, taking down the Buildkite web UI, APIs and job dispatch
    Buildkite2026
  3. IL-0471 GitHub Actions runs failed or sat queued for 9 hours after a routine deployment's pod replacement saturated capacity
    GitHub2026
  4. IL-0155 Some DOKS clusters in NYC1 saw intermittent DNS timeouts and NodeNotReady events because CoreDNS ran on shared-CPU Droplet worker nodes
    DigitalOcean2026
  5. IL-0470 Buildkite notifications were delayed after Prometheus storage filled and EKS notification workers could not autoscale
    Buildkite2026
  6. IL-0548 Buildkite's EKS Prometheus server was killed for memory again and again, so services stopped scaling and jobs lagged
    Buildkite2026
  7. IL-0435 Error rates rose as Aurora MySQL killed connections and declined queries to avoid OOM while load metrics stayed flat
    CyberAgent2026
  8. IL-0464 IONOS Managed Kubernetes control planes went unreachable for minutes at a time as their shared etcd grew and fragmented
    IONOS Cloud2026
  9. IL-0465 IONOS Container Registry image pushes and pulls failed with 504 Gateway Timeout
    IONOS Cloud2026
  10. IL-0350 Hosts in an IONOS TXL cluster lost connectivity when an InfiniBand fabric resweep failed on a full multicast table
    IONOS Cloud2026
  11. IL-0181 A traffic spike from an external community post pushed product API pods into CPU throttling under a 1-core limit, and an e-commerce product page slowed down while every pod stayed Running
    Kevin2026
  12. IL-0581 29CM product display servers lost pods and saturated Netty event loops on the first day of the Igu Week sale
    MUSINSA2025
  13. IL-0100 PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queries
    PostHog2025
  14. IL-0413 LLM inference engines timed out or ran out of resources under load in about 60% of 156 severe incidents at Microsoft
    Microsoft2025
  15. IL-0048 DNS lookups timed out in a large EKS cluster when CoreDNS hit the 1024 packets-per-second link-local limit, driven by sudo hostname lookups
    unknown2025
  16. IL-0067 Redis filled its persistent volume, corrupted its append-only file and crash-looped on every restart
    BC Government2025
  17. IL-0080 kube-dns on Mercari's GKE clusters could not keep up with peak-hour DNS query rates, and a Sev1 production outage followed when several services failed DNS resolution
    Mercari2025
  18. IL-0082 Losing one CoreDNS pod during a planned EKS node reroll overloaded the remaining CoreDNS pods, causing DNS timeouts and errors across multiple services at RebelMouse
    RebelMouse2025
  19. IL-0006 VPC subnets ran out of IP addresses because the AWS VPC CNI kept IPs on full nodes, and WARM_IP_TARGET changes then blocked pod starts after every pod deletion
    Neon2025
  20. IL-0099 User servers on 2i2c's Earthscope hub took over 15 minutes to start and kernels were killed during a workshop, because a profile option guaranteed 8 GB of memory with no limit on 32 GB nodes
    2i2c2025
  21. IL-0170 During a Lunar New Year event, traffic more than three times normal filled the nf_conntrack table on the EKS node running the Istio ingress gateway, dropping packets and taking down Backpackr's web and app services
    Backpackr2025
  22. IL-0027 All CoreDNS pods landed on one node, the node ran out of memory, and DNS for the whole service went down
    Backpackr2025
  23. IL-0085 A control plane node of Magic Pages' Kubernetes cluster on Hetzner Cloud ran out of memory, the cluster's own calls then used up the Hetzner API rate limit, and all customer websites were down for about 7.5 hours
    Magic Pages2025
  24. IL-0204 An etcd memory spike overloaded the control plane of a Render Frankfurt cluster, and DNS then failed because CoreDNS ran on the rebooted control plane nodes and could not start without the API server
    Render2024
  25. IL-0007 The AWS VPC CNI could not assign IP addresses to pods as an EKS cluster ran out of IPv4 space
    Adevinta2024
  26. IL-0232 Calico pods in the codfw WikiKube cluster hit their resource limits and crash-looped, taking nodes out one by one
    Wikimedia Foundation2024
  27. IL-0440 Pods on Cybozu's Neco Kubernetes platform got frequent no such host and i/o timeout DNS errors from dropped packets
    Cybozu2024
  28. IL-0042 Application logs stopped reaching Elasticsearch and OpenSearch for hours and were lost when OpenSearch ran out of disk space
    UK Ministry of Justice2024
  29. IL-0044 Prometheus OOMKilled in a loop while replaying its write-ahead log after a kube-prometheus-stack upgrade
    Saifeddine Rajhi2024
  30. IL-0182 Too few CoreDNS replicas on EKS were overloaded during traffic spikes, raising latency and cutting the network for task pods, and the over-provisioned, tightly spread CoreDNS pods added later blocked Karpenter consolidation
    이언철2024
  31. IL-0046 Prometheus restarted again and again because its WAL replay took longer than the startup probe allowed
    UK Ministry of Justice2024
  32. IL-0110 A Grafana Agent pod's high memory use and crash-looping Hangfire pods drove a Kubernetes node into system OOM and NotReady, twice in one day
    DFDS2024
  33. IL-0168 Simultaneous cluster auto-upgrades overloaded Scaleway's Kubernetes API, and a node authentication retry loop kept Kapsule and Kosmos nodes NotReady in FR-PAR
    Scaleway2024
  34. IL-0169 The Scaleway Kubernetes API in FR-PAR ran out of memory under simultaneous deployments, and because it also handled node authentication, Kapsule and Kosmos nodes went NotReady
    Scaleway2024
  35. IL-0202 A burst of queries and database writes exhausted connection pools and memory on Honeycomb's ingest hosts, and Kubernetes crash-loop back-off kept failed hosts offline so ingest capacity stayed too low to recover
    Honeycomb2023
  36. IL-0386 After a move to Google Cloud, Postgres connection timeouts and memcached i/o timeouts spiked on GKE nodes
    incident.io2023
  37. IL-0433 Shifting API traffic from EC2 to EKS on Fargate overloaded two CoreDNS pods and caused mass name-resolution errors
    MonotaRO2023
  38. IL-0043 Application logs missing in Kibana after a newer Fluent Bit version changed its chunk drop strategy
    UK Ministry of Justice2023
  39. IL-0016 Prometheus on the live cluster ran out of memory and was OOMKilled, and the monitoring node it ran on went NotReady
    UK Ministry of Justice2023
  40. IL-0028 Temporal History service latency rose because its pods were CPU-throttled while CPU usage read below 60 percent
    Temporal2023
  41. IL-0195 The kube-apiservers of a cluster with more than 200 nodes ran out of memory and crashed one after another under mass LIST requests from Cilium agents
    Palark2023
  42. IL-0165 Some LKE users in Frankfurt could not reach their kube-apiservers with kubectl after backend services failed under memory constraints
    Linode2023
  43. IL-0194 After a kube-apiserver restart, pod LIST requests from every pod of a Vector DaemonSet nearly tripled etcd memory and restarted all master nodes
    Palark2023
  44. IL-0070 Critical pods on core nodes evicted when Prometheus began scraping about 200 nodes after a dask-gateway scale-up
    2i2c2023
  45. IL-0500 UToronto JupyterHub logins and server starts returned 500 errors when the hub's sqlite database disk filled up
    2i2c2022
  46. IL-0140 The GKE API in us-central1 returned errors and high latency for 3 hours 19 minutes after an unexpected rise in backend traffic that client retries made worse
    Google Cloud2022
  47. IL-0206 Consul agents in GitLab.com's GKE pods were repeatedly OOM-killed, so the Rails pods lost database replica discovery and sent every query to the primary
    GitLab2022
  48. IL-0445 Chatwork's service on EKS became unstable as conntrack tables on some nodes overflowed and connections were dropped
    Chatwork2021
  49. IL-0094 Student pods filled the shared nodes of a JupyterHub cluster and starved the hub's core pods, freezing the hub
    2i2c2020
  50. IL-0093 Ingress controller pods crash-looped one after another because the leader pod had exhausted its shared memory
    UK Ministry of Justice2020
  51. IL-0176 Prometheus in a 235-node Kubernetes cluster crash-looped because over two million series exhausted host memory and the kernel OOM killer chose it first, as Burstable QoS gave it oom_score_adj 999
    LINE2020
  52. IL-0013 The conntrack table filled up on downsized GKE nodes and a high-traffic aggregation proxy saw climbing connection resets
    loveholidays2020
  53. IL-0015 A GKE cluster's /16 pod address range ran out at 256 nodes, leaving new pods Pending and node scale-ups failing
    loveholidays2020
  54. IL-0033 Uneven load across kube-dns pods and a DNS packets-per-second limit pushed service latency to 20 seconds
    Airbnb2019
  55. IL-0026 Kubernetes nodes hit SystemOOM because fluentd-sumologic log shipper pods had no memory limits
    Blue Matador2019
  56. IL-0002 CI builder pods could not get AWS IAM credentials because kubelet fell minutes behind its API rate limit
    Zalando2019
  57. IL-0076 Kubernetes masters went down after Dex filled etcd with AuthRequest objects and deleting its namespace started an endless list-and-timeout loop
    Fullstaq2019
  58. IL-0401 Prow jobs on the k8s-prow-build GKE cluster failed and merges stalled as nodes hit IOPS throttling and kernel panics
    Kubernetes project2019
  59. IL-0037 Image registry rate-limited a whole region after a crash-looping DaemonSet with imagePullPolicy Always pulled through three NAT instances
    Datadog2019
  60. IL-0064 CPU-heavy pods packed onto one Google Cloud node drove it to kernel panics, and the crash moved from node to node until the site was down
    Moonlight2019
  61. IL-0075 Kubernetes API servers pegged on CPU when the cluster grew to about 290 nodes, traced to Datadog agents polling the API
    Civis Analytics2019
  62. IL-0472 Node-to-pod traffic and DNS failed on Tinder's 605-node Flannel cluster when every node's ARP cache passed gc_thresh3
    Tinder2019
  63. IL-0001 Cluster-wide DNS outage after CoreDNS pods were OOM-killed again and again under a burst of DNS queries
    Zalando2019
  64. IL-0031 Kubernetes API server OOM-killed, leaving the Skipper ingress without routes; the talk points to a misconfigured CronJob manifest
    Zalando2018
  65. IL-0518 gitlab.freedesktop.org on GKE ran short of volume space and a planned 30-minute storage upgrade became a 4-hour outage
    freedesktop.org2018
  66. IL-0503 mybinder.org launches failed after a GKE node ran out of memory and its nginx-ingress proxy pod crash-looped
    Binder Project2017
  67. IL-0036 A reindex Job that kept failing exhausted one node after another until the whole cluster was down
    Universe2017
  68. IL-0040 Parallel Helm installs saturated small gp2 root volumes on a kops cluster and left all three workers NotReady with the kubelet or Docker down
    Saltside2017
  69. IL-0468 New JupyterHub student pods stuck Pending on MaxVolumeCount after a memory limit cut with node size unchanged
    UC Berkeley2017