Incident Library
Applied filters Root causeFaulty deploymentComponentAdmission webhookClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0088 A validating admission webhook added by a new ingress controller module version made the live-1 cluster reject every new Ingress with an x509 error
    UK Ministry of Justice2020
    Symptoms

    At 12:39 on 7 September 2020 the Cloud Platform team found that the Ingress API of the live-1 cluster was refusing every POST request. A user who provisioned a new service could not create an ingress into the cluster. The incident was declared at 12:54.

    Deploys failing
    Trigger

    Version 0.1.0 of the team's ingress controller module started creating a ValidatingWebhookConfiguration resource, and one new 0.1.0 ingress controller in the live-1 cluster failed to be created because of AWS resource limits.

    Root cause
    Faulty deployment
    Component
    Admission webhookIngress controller
    Contributing factor
    Single point of failure
    Detection

    The Ingress API was found refusing 100% of POST requests at 12:39; the incident was declared at 12:54.

    Remediation
    • removed by hand all the additional admission controllers created by module version 0.1.0
    • removed the admission webhook from the ingress controller module and released version 0.1.1
    • upgraded every ingress module on version 0.1.0 to 0.1.1
    Lessons

    Turning on the validating webhook in the shared ingress controller module made the webhook a single point of failure for every ingress controller pod in the ingress-controllers namespace: when one controller could not come up, no team could create an ingress.

    View case
Check your cluster for this

This case involves Admission webhook and Ingress controller. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Faulty deployment1 case

Component
Admission webhookIngress controller
Contributing factor
Single point of failure
Organisation
UK Ministry of Justice
  1. IL-0088 A validating admission webhook added by a new ingress controller module version made the live-1 cluster reject every new Ingress with an x509 error
    UK Ministry of Justice2020