IL-0088 UK Ministry of Justice · 2020-09-07kubernetesaws
On the Ministry of Justice Cloud Platform, teams get their ingress controllers from a shared ingress controller module. Version 0.1.0 of that module turned on the creation of a ValidatingWebhookConfiguration, so the ingress controllers' admission webhook began validating Ingress objects. Turning this on made the webhook a single point of failure for all ingress controller pods in the ingress-controllers namespace.
In the live-1 cluster one new 0.1.0 ingress controller could not be created because of AWS resource limits. From then on the webhook blocked new ingress rules: calls to the admission service of an ingress controller failed with "x509: certificate signed by unknown authority", and the Ingress API refused all POST requests. The team removed by hand all the extra admission controllers that 0.1.0 had created, took the admission webhook out of the module, released version 0.1.1 and upgraded every ingress module that was on 0.1.0. The report gives the time to repair as 3h 02m and the time to resolve as 3h 17m.
At 12:39 on 7 September 2020 the Cloud Platform team found that the Ingress API of the live-1 cluster was refusing every POST request. A user who provisioned a new service could not create an ingress into the cluster. The incident was declared at 12:54.
Creating an ingress failed in the validating admission webhook with this error: "Error from server (InternalError): error when creating "ingress.yaml": Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": Post offender-categorisation-prod-nx-controller-admission.ingress-controllers.svc:443/extensions/v1beta1/ingresses?timeout=30s: x509: certificate signed by unknown authority".
Symptom Deploys failing
Root cause Faulty deployment
The Ingress API was found refusing 100% of POST requests at 12:39; the incident was declared at 12:54.
–
Turning on the validating webhook in the shared ingress controller module made the webhook a single point of failure for every ingress controller pod in the ingress-controllers namespace: when one controller could not come up, no team could create an ingress.
IL-0188 Rolling updates on EKS returned 502 errors for about 30 seconds because Kubernetes deleted old pods as soon as new pods passed readiness, before the ALB had marked the new pod targets healthyYonghyun Kim · 2025same root cause · shared Ingress controller
IL-0086 A deployment meant for a CircleCI test Kubernetes cluster also removed Traefik and Calico resources from the production cluster, taking down the CircleCI UI and runner.circleci.comCircleCI · 2021same root cause · shared Ingress controller
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.