Incident Library
Applied filters SymptomControl plane lagFactorOverlapping failuresClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0084 Packet drops on a bare metal provider's vSwitch forced Namespace to rebuild its fra3 region and left global capacity short, and the next day slow NVMe writes on a control plane node drove the Kubernetes API server to very high CPU
    Namespace2024
    Symptoms

    From 17 October 2024, packets through the vSwitch that joined Namespace's bare metal machines at one provider in the fra3 region were dropped at random. Retries in TCP, QUIC and the Namespace control plane hid most of it, but the SLO error budget for latency targets on key interactions began to burn.

    Intermittent connection failuresHigh latencyControl plane lagService outage
    Trigger

    Random packet drops on the bare metal provider's vSwitch from 17 October 2024; for the second outage, a rate of Kubernetes changes that the slow NVMe on one control plane node could no longer keep up with.

    Root cause
    Cloud provider outage
    Component
    Kubernetes API server
    Contributing factor
    No capacity planningNo headroomOverlapping failures
    Detection

    SLO-based monitoring of latency targets showed the error budget burning while retries hid the individual packet drops.

    Remediation
    • diverted all customer traffic from fra3 to other regions
    • connected each provider machine to the zrh1 cluster over Wireguard instead of the vSwitch
    • repaved every fra3 machine with a new identity, certificates and encryption-at-rest keys, dropping regional caches to recover faster
    • cloned existing datasets in a more distributed way to speed up repaving the machines
    • added machines back to the new region in batches and returned the region to the compute pool
    • redirected workloads away from zrh1 during the second outage
    • cloned the failing control plane node's key state to a higher-capacity node that took over its role and IP address
    Action items
    • Credit all instances that ran during the outages back to customers
    • Build up backup regions and solutions so a similar event has options that do not affect customers
    Lessons

    Running hot with a delayed hardware delivery left no room to take a region out, so fixing a provider network fault turned into a global capacity shortage. After one outage the team was anchored on its cause and first looked for network problems in the second outage, which had a different cause: a control plane node whose disk could not keep up with the Kubernetes change rate.

    View case
Check your cluster for this

This case involves Kubernetes API server. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Cloud provider outage1 case

Component
Kubernetes API server
Contributing factor
No capacity planningNo headroomOverlapping failures
Organisation
Namespace
  1. IL-0084 Packet drops on a bare metal provider's vSwitch forced Namespace to rebuild its fra3 region and left global capacity short, and the next day slow NVMe writes on a control plane node drove the Kubernetes API server to very high CPU
    Namespace2024