Cases13 cases

3 to 7 of 13

1. IL-0137 Training initialisation on a 60-node NVIDIA B200 cluster took more than 8 hours instead of minutes because fragmented small random I/O from all nodes saturated the metadata service of the VAST Data NFS storage

   Lablup Inc2026

   Symptoms

   The 60-node NVIDIA B200 training configuration for the Solar Open pre-training run showed no major problem during initial validation on 2 to 4 nodes. Once the job ran at its full size of 60 nodes, training initialisation, which reads from the VAST Data storage and should have finished within minutes, took more than 8 hours, and storage throughput stayed far below the theoretical limit.

   Throughput degradation

   Trigger

   Scaling the training job from 2 to 4 nodes in validation to the full 60-node configuration.

   Root cause

   Capacity exhaustion

   Component

   Shared network filesystem

   Contributing factor

   Unrepresentative stagingMonitoring blind spot

   Detection

   The problem showed up as training initialisation taking hours after the job was scaled to 60 nodes; application logs showed only the delayed initialisation.

   Remediation

   • Sharded the Arrow training data files by rank on the application side
   • Switched the storage to asynchronous deletion instead of synchronous rm/unlink
   • Tuned readahead

   Lessons

   Storage behaviour measured on 2 to 4 nodes did not predict behaviour on 60 nodes: a metadata bottleneck of this kind appears only at production scale, so small pre-tests cannot find it. Per-node network rates can look normal while the combined access pattern of all nodes overwhelms the storage metadata path. Finding the cause took metrics from the application, scheduler, network and storage on one shared timeline; monitoring by any single team was not enough. The report concludes that GPU capacity has to be provisioned together with a matching storage path.

   View case
2. IL-0181 A traffic spike from an external community post pushed product API pods into CPU throttling under a 1-core limit, and an e-commerce product page slowed down while every pod stayed Running

   Kevin2026

   Symptoms

   An e-commerce platform suddenly became slow to respond, worst on the product lookup page, and users began reporting that the page would not open.

   CPU throttlingHigh latency

   Trigger

   A post on the external community Ppomppu linked to the platform and brought a large number of users in a short time, who called the product lookup API heavily.

   Root cause

   Capacity exhaustion

   Component

   Container CPU limitHorizontal pod autoscaler

   Contributing factor

   Monitoring blind spot

   Detection

   Users reported that the product page would not open. The dashboard showed all pods Running, and the namespace CPU usage near 100% and the CPU usage metrics pointed to throttling.

   Remediation

   • raised the HPA minReplicas from 5 to 10
   • raised the container CPU limit from 1 core to 2 cores

   Action items

   • Keep adjusting the CPU limit to a suitable level through monitoring

   Lessons

   CPU throttling is hard to notice: pods stay Running, nothing restarts and the logs show no clear error, so a pod list does not reveal it and CPU usage metrics do. Settings that had handled expected event traffic became the bottleneck as soon as one external post multiplied the traffic. The author concludes that the team needs more preparation for traffic pattern changes and more thought about resource settings.

   View case
3. IL-0170 During a Lunar New Year event, traffic more than three times normal filled the nf_conntrack table on the EKS node running the Istio ingress gateway, dropping packets and taking down Backpackr's web and app services

   Backpackr2025

   Symptoms

   During a Lunar New Year event in January 2024, a developer posted a message in Slack suspecting an outage. While several people were checking it, the problem grew into a full outage of Backpackr's web and app services, and the company decided to stop the event.

   Service outage

   Trigger

   An unexpected traffic surge, more than three times normal, during the Lunar New Year event in January 2024.

   Root cause

   Capacity exhaustion

   Component

   conntrack tableIngress controllerIstio

   Contributing factor

   Single point of failureUnreviewed defaultsMonitoring blind spot

   Detection

   A developer reported a suspected outage in Slack; the outage became total while several people were still confirming it.

   Remediation

   • stopped the Lunar New Year event

   Action items

   • Raise net.netfilter.nf_conntrack_max on all nodes to memory bytes/16384/2, and three times that on Istio nodes, applied by the nodeOptimizer DaemonSet
   • Configure Istio mesh networking so sidecars communicate directly instead of through the ingressGateway
   • Upgrade the Istio nodes from c6.xlarge (12.5 Gbps) to n-type instances with up to 25 Gbps of network bandwidth
   • Strengthen monitoring with ENA network bandwidth and packet drops, detailed Istio ingressGateway and sidecar metrics, and CPU, memory and saturation

   Lessons

   When every request in a cluster goes through one ingress gateway node, that node's connection tracking table and network bandwidth set the limit for the whole service. The default nf_conntrack table size was too small for three times normal traffic, and once it filled, packets were dropped. After the outage Backpackr strengthened its monitoring of network bandwidth, packet drops and the Istio components.

   View case
4. IL-0085 A control plane node of Magic Pages' Kubernetes cluster on Hetzner Cloud ran out of memory, the cluster's own calls then used up the Hetzner API rate limit, and all customer websites were down for about 7.5 hours

   Magic Pages2025

   Symptoms

   At around 09:33 UTC one of the three control plane nodes of the Kubernetes cluster that hosts Magic Pages went offline. Looking at the node showed that it had run out of memory.

   Service outageNode NotReadyCloud API rate limit exhausted

   Trigger

   One of the three control plane nodes ran out of memory and went offline at about 09:33 UTC; the attempts to replace it with hetzner-k3s failed while the cluster's own components were exhausting the Hetzner API rate limit.

   Root cause

   Capacity exhaustion

   Component

   Cloud controller managerPersistent volumes and stateful sets

   Contributing factor

   Unknown limitRetry stormMonitoring blind spotLimited debug accessSingle point of failureNo runbook

   Detection

   The founder found the control plane node offline at about 09:33 UTC. Monitoring alerts about unreachable customer websites started around the time the replacement attempts failed, and about half an hour of investigation identified the exhausted Hetzner API rate limit.

   Remediation

   • powered off all servers of the Kubernetes cluster to stop the API requests so the Hetzner rate limit could recover
   • waited until ratelimit-remaining had built up a buffer before starting recovery
   • restarted the control plane servers
   • started the essential Longhorn components by hand so Longhorn could heal and make storage volumes available
   • restarted the remaining websites by hand

   Action items

   • Monitor the Hetzner API call rate coming from the Kubernetes cluster nodes themselves to get early warning of high usage
   • Get a higher Hetzner API rate limit; Hetzner support was contacted
   • Investigate how to reduce the cluster's reliance on a single API, including alternatives to hetzner-k3s
   • Document and test a procedure for recovering Longhorn after a full cluster shutdown

   Lessons

   The running cluster, not only the deployment tooling, consumed the cloud provider's API quota, and in a failure it consumed it fast enough that the quota never recovered. The default limit of 3600 requests per hour was not enough for failure and recovery scenarios, and API usage from inside the cluster was not monitored. A tool that hides the provider's API errors slows diagnosis. After a full, non-graceful shutdown, Longhorn needed manual intervention to come back, which lengthened the outage.

   View case
5. IL-0110 A Grafana Agent pod's high memory use and crash-looping Hangfire pods drove a Kubernetes node into system OOM and NotReady, twice in one day

   DFDS2024

   Symptoms

   At 10:01 UTC on 2024-03-13 a Kubernetes node entered NotReady state, with CPU requests at 78% and memory requests at 82%. An alert fired in Slack at 10:03 and was picked up at 10:05. An attempt to SSH into the node at 10:13 failed because it was unresponsive; it started responding again at 10:19.

   Node NotReadyContainer crash loopOOMKilledMonitoring outage

   Trigger

   Pods of a dev container that used Hangfire to trigger jobs failed with OOM and crash-looped on a node where a Grafana Agent pod was already using a lot of memory.

   Root cause

   Capacity exhaustion

   Component

   Telemetry agentContainer memory limitApplication database

   Contributing factor

   Excessive blast radiusNo resource requests or limitsHarmful manual actionMonitoring blind spot

   Detection

   A Slack alert fired two minutes after the node went NotReady and was picked up two minutes later; the second node's NotReady at 15:03 also raised a Slack alert.

   Remediation

   • marked the EC2 instance unhealthy so a new node started
   • cordoned and force-drained the NotReady node and terminated its EC2 instance
   • removed the new LimitRange from the Grafana Agent namespace and restarted the Grafana Agent pod
   • scaled the dev container to 0 replicas, after which EC2 network traffic returned to normal
   • after the second NotReady, scaled the dev container to 0 replicas again and added a new node
   • emailed the team that owns the dev container, which confirmed it was working on a fix

   Action items

   • Move grafana-agent to the monitoring node group
   • Add alerts for when bandwidth across the cluster reaches a certain threshold
   • Investigate whether memory limits can be set on the grafana-agent Helm chart
   • Investigate grafana-agent memory usage
   • Ask Grafana support whether grafana-agent should use a PVC
   • Consider pushing node logs to Loki with limited retention
   • Create a wiki page with common metrics and log queries to check when investigating issues with Kubernetes nodes

   Lessons

   Dev workloads mixed with production let one crash-looping dev container take down nodes that production also used, and a monitoring agent outside its dedicated node group added to the memory pressure. Scaling the suspect container back up before its owners had fixed it brought the failure back on the replacement node. The team also noted that action items from earlier post-mortems need follow-up, and that knowledge of useful CloudWatch metrics had not been shared, which made the diagnosis harder.

   View case

By root cause1