Cases3 cases

1 to 3 of 3

1. IL-0137 Training initialisation on a 60-node NVIDIA B200 cluster took more than 8 hours instead of minutes because fragmented small random I/O from all nodes saturated the metadata service of the VAST Data NFS storage

   Lablup Inc2026

   Symptoms

   The 60-node NVIDIA B200 training configuration for the Solar Open pre-training run showed no major problem during initial validation on 2 to 4 nodes. Once the job ran at its full size of 60 nodes, training initialisation, which reads from the VAST Data storage and should have finished within minutes, took more than 8 hours, and storage throughput stayed far below the theoretical limit.

   Throughput degradation

   Trigger

   Scaling the training job from 2 to 4 nodes in validation to the full 60-node configuration.

   Root cause

   Capacity exhaustion

   Component

   Shared network filesystem

   Contributing factor

   Unrepresentative stagingMonitoring blind spot

   Detection

   The problem showed up as training initialisation taking hours after the job was scaled to 60 nodes; application logs showed only the delayed initialisation.

   Remediation

   • Sharded the Arrow training data files by rank on the application side
   • Switched the storage to asynchronous deletion instead of synchronous rm/unlink
   • Tuned readahead

   Lessons

   Storage behaviour measured on 2 to 4 nodes did not predict behaviour on 60 nodes: a metadata bottleneck of this kind appears only at production scale, so small pre-tests cannot find it. Per-node network rates can look normal while the combined access pattern of all nodes overwhelms the storage metadata path. Finding the cause took metrics from the application, scheduler, network and storage on one shared timeline; monitoring by any single team was not enough. The report concludes that GPU capacity has to be provisioned together with a matching storage path.

   View case
2. IL-0129 Throughput of the 384-GPU SmolLM3 pretraining run collapsed because shared WekaFS storage evicted dataset shards to S3, then kept dropping on local NVMe because the nanotron dataloader built an index that grew with the step count

   Hugging Face2025

   Symptoms

   Within hours of launching the full SmolLM3 pretraining run (3B parameters, 11T tokens, 384 H100 GPUs on 48 nodes), training throughput fell steeply, with repeated sharp drops. The team expected a steady 13,500 to 14,000 tokens per second, and none of the throughput ablations run beforehand on the same 48 nodes had shown drops.

   Throughput degradation

   Trigger

   Launching the full run with the whole ~24 TB dataset on shared WekaFS storage and the real step count for 11T tokens, both far larger than in the ablation runs.

   Root cause

   Capacity exhaustion

   Component

   Shared network filesystemTraining data loader

   Contributing factor

   Overlapping failuresUnrepresentative stagingUnverified assumption

   Detection

   The team watched training throughput in tokens per second and saw it fall within hours of launch; node-monitoring metrics then tied the large fall to spikes in disk read latency.

   Remediation

   • moved the training data from WekaFS to the local NVMe RAID scratch storage on each node
   • copied the dataset to replacement nodes from a healthy node with fpsync, cutting the copy from 3 hours to 1.5 hours
   • kept a spare node with the dataset preloaded in the Slurm reservation to swap in when a node failed
   • reproduced the drops on a single node and with step counts of about 3M and 32k to rule out hardware and isolate the step count
   • replaced nanotron's nanosets dataloader with the TokenizedBytes dataloader copied from the SmolLM2 setup, which removed the drops
   • pre-shuffled the tokenized sequences offline for each epoch with different seeds (about 1 hour) to fix the noisier loss from TokenizedBytes

   Lessons

   Ablations on data subsets and a short step horizon did not exercise either cause: a shared cache-and-evict filesystem near its capacity, and a dataloader index that grows with the step count. Reproducing the drops on one node separated hardware from software, and varying only the step count pointed at the dataloader. The team chose proven replacements over repairs under deadline pressure, taking an existing dataloader instead of fixing nanosets and offline shuffling instead of changing the dataloader, and warns that too many such shortcuts leave a patchwork system that is hard to maintain.

   View case
3. IL-0100 PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queries

   PostHog2025

   Symptoms

   On 21 October 2025 at 21:45 UTC a change to timeout handling was deployed to the feature flags service, and two minutes later automated monitoring reported higher error rates. The change was rolled back by 21:49, but error rates stayed high: about 38% of flag evaluation requests in the US datacenter returned errors. At 22:30 Redis metrics showed memory exhaustion on ElastiCache, at 22:35 Postgres connections spiked, and at 22:45 the team found a massive data transfer from Postgres into Redis. The main PostHog application was affected as well, because it shares the Redis instance with the flags service.

   Sustained high node CPUHigh error rateHigh latencyContainer crash loopDatabase overload

   Trigger

   On 21 October a deployment meant to reduce timeout errors (PR #39821); on 24 October the deployment of IP-based rate limiting (PR #40074) that saw all traffic as coming from the load balancer's IP; on 28 October an unrelated routine deployment that rolled the feature flag pods in the US region.

   Root cause

   Capacity exhaustion

   Component

   Node poolPod scheduling and placementIn-memory cacheApplication database

   Contributing factor

   No alert on limitRetry stormIneffective rollbackSingle point of failureNo staged rolloutUnrepresentative staging

   Detection

   On 21 October automated monitoring detected higher error rates two minutes after the deploy. On 24 October no alert existed for 429 errors, and the team learned of the incident from customer reports that began at 18:10 UTC, a 62-minute detection delay. On 28 October on-call received a high error count alert. On 29 October the incident was reported with higher error rates and latency. No alert existed for CPU pressure, so the CPU saturation behind all three incidents was not seen directly.

   Remediation

   • rolled back the timeout handling change within 2 minutes on 21 October, which did not stop the errors
   • throttled connections and Redis writes on 21 October
   • halted the rollout on 28 October, scaled the US flags pods to zero until Redis recovered, then scaled them back gradually
   • rolled back hardware changes on 30 October at 00:03 UTC, which removed most errors but not the latency
   • increased the pod resource requests and adjusted the Kubernetes fleet size so fewer pods ran on each node
   • added retry limits to Envoy between the load balancer and the service
   • moved database connection timeouts and other critical settings to runtime configuration and raised the timeout values
   • added per-node and per-pod CPU alerts at 80% sustained for 5 minutes that page on-call

   Action items

   • Form a strike team from the flags, ingestion and infrastructure teams to review the application and infrastructure for remaining bottlenecks
   • Investigate separating the flags Redis instance from the application Redis instance
   • Before re-enabling ArgoCD sync, align the flags service chart configuration with the values in ArgoCD and define the deployment and Redis strategy
   • Make the feature flags team hard code-owners for flags-related code
   • Build a dashboard of key flag metrics with runbook links and update the feature flag runbooks
   • Add missing alerts on existing service and infrastructure metrics
   • Make readiness checks validate dependencies that degrade under load, for example by pinging the database
   • Rate-limit cache population so it cannot overwhelm Redis
   • Monitor connection pools with automatic throttling near exhaustion and limit concurrent connections
   • Build a production-scale load testing framework
   • Build progressive rollout infrastructure to limit blast radius
   • Document Postgres-to-Redis data flows and write runbooks for data transfer storms

   Lessons

   Nodes that are small relative to pod requests let the scheduler pack many pods on each node, and the resulting CPU saturation showed up three different ways: excessive parallelism, connection pool exhaustion and slow queries. Treating each symptom separately let the problem return; right-sizing the pods and nodes fixed all three. CPU must be alerted on from the start. Unbounded Envoy retries and unbounded cache population amplified the overload, and a Redis instance shared with the main application spread a flags problem to the main application. A rollback did not end the incident, because the data transfer and resource exhaustion it had started kept running.

   View case

By root cause1