Incident Library
Applied filters FactorSingle point of failureOrganisationUK Ministry of JusticeClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0088 A validating admission webhook added by a new ingress controller module version made the live-1 cluster reject every new Ingress with an x509 error
    UK Ministry of Justice2020
    Symptoms

    At 12:39 on 7 September 2020 the Cloud Platform team found that the Ingress API of the live-1 cluster was refusing every POST request. A user who provisioned a new service could not create an ingress into the cluster. The incident was declared at 12:54.

    Deploys failing
    Trigger

    Version 0.1.0 of the team's ingress controller module started creating a ValidatingWebhookConfiguration resource, and one new 0.1.0 ingress controller in the live-1 cluster failed to be created because of AWS resource limits.

    Root cause
    Faulty deployment
    Component
    Admission webhookIngress controller
    Contributing factor
    Single point of failure
    Detection

    The Ingress API was found refusing 100% of POST requests at 12:39; the incident was declared at 12:54.

    Remediation
    • removed by hand all the additional admission controllers created by module version 0.1.0
    • removed the admission webhook from the ingress controller module and released version 0.1.1
    • upgraded every ingress module on version 0.1.0 to 0.1.1
    Lessons

    Turning on the validating webhook in the shared ingress controller module made the webhook a single point of failure for every ingress controller pod in the ingress-controllers namespace: when one controller could not come up, no team could create an ingress.

    View case
  2. IL-0093 Ingress controller pods crash-looped one after another because the leader pod had exhausted its shared memory
    UK Ministry of Justice2020
    Symptoms

    At 10:43 on 14 August 2020 two of the six ingress controller pods in the cluster were in a crash loop. Restarting those pods did not fix them.

    Container crash loop
    Root cause
    Capacity exhaustion
    Component
    Ingress controller
    Contributing factor
    Harmful manual actionSingle point of failure
    Detection

    Two of the six ingress controller pods were seen crash-looping at 10:43; the incident was declared at 11:01.

    Remediation
    • restarted the leader ingress controller pod so the other replicas could connect and get the latest nginx configuration
    View case
Check your cluster for this

These 2 cases involve Ingress controller and Admission webhook. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause2

Faulty deployment1 case

Component
Admission webhookIngress controller
Contributing factor
Single point of failure
Organisation
UK Ministry of Justice
  1. IL-0088 A validating admission webhook added by a new ingress controller module version made the live-1 cluster reject every new Ingress with an x509 error
    UK Ministry of Justice2020

Capacity exhaustion1 case

Component
Ingress controller
Contributing factor
Harmful manual actionSingle point of failure
Organisation
UK Ministry of Justice
  1. IL-0093 Ingress controller pods crash-looped one after another because the leader pod had exhausted its shared memory
    UK Ministry of Justice2020