Incident Library
Applied filters ComponentPolicy engineFactorDeferred upgradesOrganisationDFDSClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0107 Kyverno stuck in a crash loop behind a webhook with failurePolicy fail blocked every change to Kubernetes Service objects for 88 minutes
    DFDS2022
    Symptoms

    At 11:38 UTC on 3 August 2022 an end-user team told the platform team that applying a Service resource to one of the Kubernetes clusters failed.

    Deploys failingContainer crash loop
    Trigger

    A certificate chain in a third-party application expired, through no action of DFDS, and the outdated Kyverno installation got stuck in a crash loop.

    Root cause
    Certificate or credential expiry
    Component
    Policy engineAdmission webhook
    Contributing factor
    Deferred upgradesUnreviewed defaultsMonitoring blind spotIgnored early signal
    Detection

    An end-user team alerted the platform team at 11:38 UTC that applying a Service resource failed; the platform team started looking into it at 11:50 UTC.

    Remediation
    • reproduced the issue in a sandbox cluster and deployed the fix there first
    • updated Kyverno to a newer Helm chart version in the affected production cluster
    Action items
    • Implement monitoring on Kyverno
    • Create internal documentation on the Kyverno setup
    • Investigate the exact certificate issue in the third-party application
    • Consider changing the failurePolicy on ClusterPolicy resources going forward
    Lessons

    Operational issues should be added to the 'Ops Day to Day' board from the start.

    View case
Check your cluster for this

This case involves Admission webhook and Policy engine. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Certificate or credential expiry1 case

Component
Admission webhookPolicy engine
Contributing factor
Unreviewed defaultsIgnored early signalMonitoring blind spotDeferred upgrades
Organisation
DFDS
  1. IL-0107 Kyverno stuck in a crash loop behind a webhook with failurePolicy fail blocked every change to Kubernetes Service objects for 88 minutes
    DFDS2022