Incident Library
Applied filters Root causeCloud provider outageComponentkubeletFactorLimited debug accessClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0469 The managed Azure Kubernetes API server for the data100 hub stopped answering, so the stopped hub pod was not recreated
    UC Berkeley2019
    Symptoms

    On 25 February 2019 the Kubernetes API server of the managed Azure cluster running the data100 hub became unreachable. kubectl failed for every operation and requests to create new resources failed, so when the hub pod stopped, no new one was created and users saw a proxy error message.

    Control plane unresponsiveService outage
    Root cause
    Cloud provider outage
    Component
    AKS control planeKubernetes API serverkubelet
    Contributing factor
    Limited debug access
    Detection

    Infrastructure staff were notified in Slack at 11:34 and found that kubectl failed for all operations while the hub proxy was up.

    Remediation
    • Created a new cluster and requested a new NFS server in its resource group
    • Detached the data volumes from the old NFS server and moved them to the new cluster's resource group
    • Imported the ZFS pool into the new NFS server
    • Created the staging hub with helm and updated DNS
    • Created the prod hub with helm and updated DNS
    Action items
    • Monitor the API server endpoint remotely and alert when it is down
    View case
Check your cluster for this

This case involves AKS control plane and Kubernetes API server. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Cloud provider outage1 case

Component
AKS control planeKubernetes API serverkubelet
Contributing factor
Limited debug access
Organisation
UC Berkeley
  1. IL-0469 The managed Azure Kubernetes API server for the data100 hub stopped answering, so the stopped hub pod was not recreated
    UC Berkeley2019