Incident Library
Applied filters Root causeDependency failureClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases13 cases

13 to 13 of 13

  1. IL-0073 Intermittent Kafka connectivity overloaded Docker daemons, set off a node rescheduling cascade and poisoned the Consul gossip mesh
    Target2019sev1
    Symptoms

    An upgrade of the network subsystem under Target's datacenter OpenStack, meant to cause only a brief disruption, ran into problems and left network connectivity disrupted for several hours. Kafka, which runs on that OpenStack, became intermittently available.

    Node NotReadyLoad imbalanceHigh latencyCredentials unavailable
    Trigger

    A network subsystem upgrade in the datacenter OpenStack infrastructure disrupted connectivity for several hours and made Kafka intermittently available.

    Root cause
    Dependency failure
    Component
    Container runtimeService discoveryTelemetry agentCredential broker for pods
    Contributing factor
    Excessive blast radiusAutomation amplificationSingle point of failureUnpatched known bug
    Remediation
    • enabled gossip encryption on Consul so messages from the poisoned mesh were rejected
    • stabilized the deployment engine and redeployed the development environment with encryption, terminating instances without it
    • upgraded Consul in place to 1.2.3 and then to 1.3.0
    Action items
    • Reduce how much of an outage a single Docker daemon can cause, together with running smaller and more numerous clusters
    Lessons

    Smaller and more numerous clusters limited the damage: the smaller development clusters and production were not hit the same way as the one large cluster, and dev and prod on one cluster would have been far worse. A Docker daemon shared by many workloads is a brittle failure point. Per-workload logging and metrics sidecars were judged the right design; a shared cluster-level agent would almost certainly have made the problem worse and harder to fix.

    View case
Check your cluster for this

These 13 cases involve InfiniBand fabric and Data center power and cooling. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Dependency failure13 cases

Component
InfiniBand fabric5Data center power and cooling3NCCL3Node operating system image3GPU driver2Node pool2PyTorch2Admission webhook1Cloud block storage1Container runtime1Credential broker for pods1Application database1Authoritative DNS servers and records1etcd1GPU hardware and memory1Liveness and readiness probes1Istio1Object storage1Container registry and its credentials1Service discovery1Shared network filesystem1Telemetry agent1VPC DNS resolver1
Contributing factor
Monitoring blind spot4Automation amplification3Unreviewed defaults3Single point of failure3Excessive blast radius2Unknown limit2No runbook2Overlapping failures2Unverified assumption1Unpatched known bug1Limited debug access1No alert on limit1No headroom1Partial rollout1
Organisation
Linode2Crusoe Cloud1Databricks1FREE NOW1Google Cloud1Imbue1Lambda1Meta AI1Nebius1Target1Tetrate (unnamed U.S. federal customer)1Yonghyun Kim1
  1. IL-0133 A storm shut down cooling at Nebius's us-central1 data center, and a disk hot-plug bug kept Managed Kubernetes control planes down for hours after power returned
    Nebius2026
  2. IL-0023 A training run crashed with an NCCL timeout after one InfiniBand port went down once and recovered
    Databricks2026
  3. IL-0134 A power event at Lambda's AUS01 data center rebooted 246 GPU nodes, all spine switches and the VAST storage, failing customer workloads and leaving stale NFS mounts on VMs
    Lambda2026
  4. IL-0187 A brief Spring Cloud Config Server downtime failed every client application's liveness probe, because the actuator health check included the config server, and Kubernetes restarted the pods in a kill loop
    Yonghyun Kim2025
  5. IL-0162 LKE users could not reach their cluster control planes or the LKE Dashboard because an earlier database update made the internal LKE DNS service hang
    Linode2025
  6. IL-0103 During an EKS node AMI update, Istio sidecars crash-looped because hundreds of pods restarting at once overloaded the external certificate authority behind istiod
    Tetrate (unnamed U.S. federal customer)2025
  7. IL-0159 Docker Hub rate-limited Linode's IPv6 address space with 429 Too Many Requests, breaking image pulls for LKE and LKE-E clusters
    Linode2025
  8. IL-0025 Failed boots, InfiniBand link flapping, NCCL hangs and throughput sags while bringing up a 4,088-GPU H100 training cluster
    Imbue2024
  9. IL-0021 GKE GPU nodes on Container-Optimized OS could not download NVIDIA drivers (403 Forbidden) and came up without working GPUs
    Google Cloud2024
  10. IL-0135 A power source switch during UPS maintenance cut power to a row of GPU servers and shut down Crusoe Cloud VMs in us-northcentral1
    Crusoe Cloud2023
  11. IL-0126 The OPT-175B training run restarted more than 40 times in two weeks as cloud GPU nodes failed with ECC errors, replacement nodes came back faulty, and checkpoint downloads hung
    Meta AI2021
  12. IL-0039 New Kubernetes worker nodes could not join any cluster because a package URL hardcoded in kops nodeup broke
    FREE NOW2019
  13. IL-0073 Intermittent Kafka connectivity overloaded Docker daemons, set off a node rescheduling cascade and poisoned the Consul gossip mesh
    Target2019