IL-0469 UC Berkeley · 2019-02-25kubernetesazure

The managed Azure Kubernetes API server for the data100 hub stopped answering, so the stopped hub pod was not recreated

At 11:21 on 25 February 2019 the API server of the managed Kubernetes cluster on Azure that ran UC Berkeley's data100 hub stopped answering. The hub proxy stayed up, but kubectl failed for all operations, and when the hub pod was stopped Kubernetes could not create a replacement, so users got a proxy error. At 11:57 staff opened a C ticket in the cloud provider's portal. C tickets have 8-hour response times, so at 12:28 staff emailed contacts at the cloud provider to ask for escalation, and the provider called back between 12:43 and 12:49.

Meanwhile staff moved the hub to a new cluster. At 12:40 they decided to import the ZFS pool into a new NFS server in the new cluster instead of moving the old NFS server between clusters. The cluster was created at 13:00, the data volumes were detached from the old server and moved into the new cluster's resource group at 13:10, and the ZFS pool was imported at 13:20. helm brought up the staging hub by 13:34 and the prod hub by 13:41, with DNS updated after each, and at 14:48 the prod hub obtained an SSL certificate from letsencrypt so students could connect. On 15 March the cloud provider replied that problems with the infrastructure hosting the cluster had made the kubelet on the master stop responding, and that this was not related to region stability.

Severity: –
Scope: The data100 hub: requests to create resources failed and, once the hub pod stopped, users saw a proxy error until the hub was rebuilt on a new cluster and could get its SSL certificate
Users affected: Students using the data100 hub
Region: –
Time to detect: 13m
Time to resolve: 3h 27m

Symptoms

On 25 February 2019 the Kubernetes API server of the managed Azure cluster running the data100 hub became unreachable. kubectl failed for every operation and requests to create new resources failed, so when the hub pod stopped, no new one was created and users saw a proxy error message.

Infrastructure staff were notified in Slack at 11:34 and found the hub proxy still up while the API server could not be reached. The cloud provider's status page showed no other reports. The start time, 11:21, was worked out only afterwards from the cloud provider's monitoring metrics.

Symptom Control plane unresponsive Service outage

Root cause

Root cause Cloud provider outage

Component AKS control plane Kubernetes API server kubelet

Contributing factor Limited debug access

Trigger: –

Detection

Infrastructure staff were notified in Slack at 11:34 and found that kubectl failed for all operations while the hub proxy was up.

Remediation

Created a new cluster and requested a new NFS server in its resource group
Detached the data volumes from the old NFS server and moved them to the new cluster's resource group
Imported the ZFS pool into the new NFS server
Created the staging hub with helm and updated DNS
Created the prod hub with helm and updated DNS

Action items

Monitor the API server endpoint remotely and alert when it is down

Lessons

–

Sources

https://github.com/berkeley-dsep-infra/datahub/blob/staging/docs/incidents/2019-02-25-k8s-api-server-down.qmd · github

Related cases

IL-0084 Packet drops on a bare metal provider's vSwitch forced Namespace to rebuild its fra3 region and left global capacity short, and the next day slow NVMe writes on a control plane node drove the Kubernetes API server to very high CPUNamespace · 2024
same root cause · shared Kubernetes API server
IL-0145 A power sag and chiller failure in one Australia East availability zone took down the AKS control plane's compute and its non-zone-redundant SQL Database operation queue, blocking AKS create, update and delete operations in Australia East and Australia SoutheastMicrosoft Azure · 2023
same root cause · shared AKS control plane
IL-0505 kubectl logs failed on GKE with "No SSH tunnels currently open" and mybinder.org build logs stopped reaching usersBinder Project · 2019
same root cause · shared Kubernetes API server

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, AKS control plane, Kubernetes API server, kubelet among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0469
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0469/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.