Incident Library
Applied filters Root causeConfiguration changeComponentAdmission webhookFactorMonitoring blind spotClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0108 An Alcide admission controller applied to the production cluster during a workshop made create and update requests on Deployments time out
    DFDS2020
    Symptoms

    At 14:41 CET on 4 November 2020 users reported timeouts when deploying to Hellman, the production Kubernetes cluster. The timeouts were confirmed at 15:04.

    Deploys failingControl plane lag
    Trigger

    During an Alcide workshop that began at 13:00 CET, an Alcide admission controller was applied to the production cluster at 13:13 in the belief that it would not disrupt anything.

    Root cause
    Configuration change
    Component
    Admission webhookKubernetes API server
    Contributing factor
    No configuration testsUnverified assumptionMonitoring blind spot
    Detection

    Users reported timeouts when deploying to the Hellman cluster at 14:41 CET; the timeouts were verified at 15:04.

    Remediation
    • toggled the Alcide admission controller to verify that it caused the timeouts
    • permanently deleted the Alcide admission controller
    Action items
    • Monitor Kubernetes deployments for timeouts
    • Monitor 5xx status codes from the Kubernetes API server
    Lessons

    New components are applied in sandbox clusters, not in production. The team needs enough Alcide knowledge to maintain the component until it is fully deprecated.

    View case
  2. IL-0008 An unresponsive OPA validating webhook crash-looped the API server during a GKE master upgrade, and node auto-repair then recreated every node
    Jetstack2019
    Symptoms

    Jetstack was upgrading the master of a development cluster that many teams use to test their apps during the working day, a regional Google Kubernetes Engine (GKE) cluster in europe-west1. The upgrade ran through Jetstack's GKE Terraform pipeline and did not finish before the 20-minute Terraform timeout, while the GKE console still showed the cluster as upgrading.

    Upgrade stallControl plane unresponsiveNode recreation loopService outage
    Trigger

    A planned master upgrade of the shared development cluster. The same upgrade had been done on another pre-production environment earlier that day, and no API downtime was expected.

    Root cause
    Configuration change
    Component
    Admission webhookGKE control planeKubernetes API serverkubelet
    Contributing factor
    Excessive blast radiusNo health probesMonitoring blind spotAutomation amplification
    Detection

    The first sign was the master upgrade missing the 20-minute Terraform timeout; re-running the pipeline returned the kube-apiserver unhealthy error. The cause was identified with the help of Google Support.

    Remediation
    • Deleted the OPA ValidatingAdmissionWebhook resource during a window of API server access
    Action items
    • Limit the OPA webhook to the namespaces where the policy applies and development teams have access
    • Enable the webhook only for Ingress and Service resources, the only resources the policy validates
    • Add a liveness probe so OPA is restarted when it stops responding, and update the OPA documentation
    • Follow the open upstream Kubernetes issue about this behaviour
    Lessons

    With alerting on API server response times, the team might have seen CREATE and UPDATE requests slow down across the board as soon as the OPA-backed webhook was first deployed. Every workload needs probes: OPA looked so simple to deploy that the team skipped the Helm chart, which adds a livenessProbe for the admission controller containers among other adjustments to the tutorial configuration. Others had hit the same problem, and an upstream issue to improve it was still open.

    View case
Check your cluster for this

These 2 cases involve Admission webhook and Kubernetes API server. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Configuration change2 cases

Component
Admission webhook2Kubernetes API server2GKE control plane1kubelet1
Contributing factor
Monitoring blind spot2Unverified assumption1Automation amplification1Excessive blast radius1No configuration tests1No health probes1
Organisation
DFDS1Jetstack1
  1. IL-0108 An Alcide admission controller applied to the production cluster during a workshop made create and update requests on Deployments time out
    DFDS2020
  2. IL-0008 An unresponsive OPA validating webhook crash-looped the API server during a GKE master upgrade, and node auto-repair then recreated every node
    Jetstack2019