Cases3 cases

1 to 3 of 3

1. IL-0084 Packet drops on a bare metal provider's vSwitch forced Namespace to rebuild its fra3 region and left global capacity short, and the next day slow NVMe writes on a control plane node drove the Kubernetes API server to very high CPU

   Namespace2024

   Symptoms

   From 17 October 2024, packets through the vSwitch that joined Namespace's bare metal machines at one provider in the fra3 region were dropped at random. Retries in TCP, QUIC and the Namespace control plane hid most of it, but the SLO error budget for latency targets on key interactions began to burn.

   Intermittent connection failuresHigh latencyControl plane lagService outage

   Trigger

   Random packet drops on the bare metal provider's vSwitch from 17 October 2024; for the second outage, a rate of Kubernetes changes that the slow NVMe on one control plane node could no longer keep up with.

   Root cause

   Cloud provider outage

   Component

   Kubernetes API server

   Contributing factor

   No capacity planningNo headroomOverlapping failures

   Detection

   SLO-based monitoring of latency targets showed the error budget burning while retries hid the individual packet drops.

   Remediation

   • diverted all customer traffic from fra3 to other regions
   • connected each provider machine to the zrh1 cluster over Wireguard instead of the vSwitch
   • repaved every fra3 machine with a new identity, certificates and encryption-at-rest keys, dropping regional caches to recover faster
   • cloned existing datasets in a more distributed way to speed up repaving the machines
   • added machines back to the new region in batches and returned the region to the compute pool
   • redirected workloads away from zrh1 during the second outage
   • cloned the failing control plane node's key state to a higher-capacity node that took over its role and IP address

   Action items

   • Credit all instances that ran during the outages back to customers
   • Build up backup regions and solutions so a similar event has options that do not affect customers

   Lessons

   Running hot with a delayed hardware delivery left no room to take a region out, so fixing a provider network fault turned into a global capacity shortage. After one outage the team was anchored on its cause and first looked for network problems in the second outage, which had a different cause: a control plane node whose disk could not keep up with the Kubernetes change rate.

   View case
2. IL-0469 The managed Azure Kubernetes API server for the data100 hub stopped answering, so the stopped hub pod was not recreated

   UC Berkeley2019

   Symptoms

   On 25 February 2019 the Kubernetes API server of the managed Azure cluster running the data100 hub became unreachable. kubectl failed for every operation and requests to create new resources failed, so when the hub pod stopped, no new one was created and users saw a proxy error message.

   Control plane unresponsiveService outage

   Root cause

   Cloud provider outage

   Component

   AKS control planeKubernetes API serverkubelet

   Contributing factor

   Limited debug access

   Detection

   Infrastructure staff were notified in Slack at 11:34 and found that kubectl failed for all operations while the hub proxy was up.

   Remediation

   • Created a new cluster and requested a new NFS server in its resource group
   • Detached the data volumes from the old NFS server and moved them to the new cluster's resource group
   • Imported the ZFS pool into the new NFS server
   • Created the staging hub with helm and updated DNS
   • Created the prod hub with helm and updated DNS

   Action items

   • Monitor the API server endpoint remotely and alert when it is down

   View case
3. IL-0505 kubectl logs failed on GKE with "No SSH tunnels currently open" and mybinder.org build logs stopped reaching users

   Binder Project2019

   Symptoms

   On 20 February 2019 at 08:03 CET, `kubectl logs -f <pod>` stopped working for every pod in the mybinder.org cluster on GKE. Because pod logs could not be read, build logs were no longer forwarded to users, although the builds themselves ran normally.

   Monitoring outageDeploys failing

   Root cause

   Cloud provider outage

   Component

   GKE control planeKubernetes API server

   Contributing factor

   Limited debug access

   Remediation

   • Cordoned node gke-prod-a-users-b921bb88-rd5d so it would be removed from the pool once empty
   • Drained the node and removed it from the pool, after which kubectl logs worked again

   Action items

   • Test that kubectl logs works and notify when it does not
   • Add the command for retrieving API server logs to the SRE guide
   • Describe in the SRE guide how to identify suspicious node networking status

   Lessons

   The root cause was very difficult to identify, partly because it appears to sit in the Kubernetes master or the GKE service itself. Capturing `kubectl describe node` for the suspicious node would have allowed a comparison with healthy nodes, and a node that reports no ExternalIP or InternalIP in its status was likely a direct symptom of the problem and can be used as a diagnostic in the future.

   View case

By root cause1