Incident Library
Applied filters Root causeConfiguration changeComponentKubernetes API serverClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases6 cases

6 to 6 of 6

  1. IL-0008 An unresponsive OPA validating webhook crash-looped the API server during a GKE master upgrade, and node auto-repair then recreated every node
    Jetstack2019
    Symptoms

    Jetstack was upgrading the master of a development cluster that many teams use to test their apps during the working day, a regional Google Kubernetes Engine (GKE) cluster in europe-west1. The upgrade ran through Jetstack's GKE Terraform pipeline and did not finish before the 20-minute Terraform timeout, while the GKE console still showed the cluster as upgrading.

    Upgrade stallControl plane unresponsiveNode recreation loopService outage
    Trigger

    A planned master upgrade of the shared development cluster. The same upgrade had been done on another pre-production environment earlier that day, and no API downtime was expected.

    Root cause
    Configuration change
    Component
    Admission webhookGKE control planeKubernetes API serverkubelet
    Contributing factor
    Excessive blast radiusNo health probesMonitoring blind spotAutomation amplification
    Detection

    The first sign was the master upgrade missing the 20-minute Terraform timeout; re-running the pipeline returned the kube-apiserver unhealthy error. The cause was identified with the help of Google Support.

    Remediation
    • Deleted the OPA ValidatingAdmissionWebhook resource during a window of API server access
    Action items
    • Limit the OPA webhook to the namespaces where the policy applies and development teams have access
    • Enable the webhook only for Ingress and Service resources, the only resources the policy validates
    • Add a liveness probe so OPA is restarted when it stops responding, and update the OPA documentation
    • Follow the open upstream Kubernetes issue about this behaviour
    Lessons

    With alerting on API server response times, the team might have seen CREATE and UPDATE requests slow down across the board as soon as the OPA-backed webhook was first deployed. Every workload needs probes: OPA looked so simple to deploy that the team skipped the Helm chart, which adds a livenessProbe for the admission controller containers among other adjustments to the tutorial configuration. Others had hit the same problem, and an upstream issue to improve it was still open.

    View case
Check your cluster for this

These 6 cases involve Kubernetes API server and Admission webhook. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Configuration change6 cases

Component
Kubernetes API server6Admission webhook2Client-side DNS resolver and cache1etcd1GKE control plane1kubelet1RBAC and service accounts1Container registry and its credentials1Telemetry agent1
Contributing factor
Monitoring blind spot4Excessive blast radius3Automation amplification2Limited debug access2Unverified assumption1Cache-masked failure1No configuration tests1No health probes1No safeguard on destructive change1No staged rollout1Unrepresentative staging1
Organisation
OpenAI2Aqua Security1DFDS1DigitalOcean1Jetstack1
  1. IL-0005 A new telemetry service overloaded the Kubernetes API servers of large clusters and broke DNS-based service discovery for all OpenAI services
    OpenAI2024
  2. IL-0118 A global change to Kubernetes namespace labels made the networking layer recompute metadata and overwhelmed the control planes of three of OpenAI's largest GPU clusters
    OpenAI2024
  3. IL-0152 An incorrect configuration applied during maintenance made an internal infrastructure cluster automatically delete the control planes of 200 DOKS clusters in FRA1
    DigitalOcean2024
  4. IL-0068 Anonymous access to the Kubernetes API server used to plant a ClusterRoleBinding backdoor and a cryptominer DaemonSet
    Aqua Security2023
  5. IL-0108 An Alcide admission controller applied to the production cluster during a workshop made create and update requests on Deployments time out
    DFDS2020
  6. IL-0008 An unresponsive OPA validating webhook crash-looped the API server during a GKE master upgrade, and node auto-repair then recreated every node
    Jetstack2019