IL-0090 UK Ministry of Justice · 2021-06-09kubernetes
The ModSec ingress controller registers a validating admission webhook, which the Kubernetes API server calls before it accepts any Ingress create or change. After the module was upgraded to v3.33.0, the API server called the webhook at the service name modsec01-nx-modsec-admission, but the TLS certificate the webhook served was valid only for modsec01-nx-controller-admission. The API server therefore could not call the webhook, and every Ingress create or change was rejected, including those in all user application deployments.
The team rolled the ModSec ingress controller module back to v0.0.7. The incident was repaired at 13:46, 31 minutes after the first report, and resolved at 13:58. The review asks why the test cluster did not show the problem, and notes that neither the old nor the new ModSec module version worked on the cluster named live.
At 13:15 on 9 June 2021 a user reported in the #ask-cloud-platform channel that deploying a UAT application failed on its Ingress resource.
The error was: kind Ingress: Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": Post https://modsec01-nx-modsec-admission.ingress-controllers.svc:443/networking/v1beta1/ingresses?timeout=10s: x509: certificate is valid for modsec01-nx-controller-admission, modsec01-nx-controller-admission.ingress-controllers.svc, not modsec01-nx-modsec-admission.ingress-controllers.svc
Every Ingress API call was blocked. No new Ingress could be created and no change to an existing Ingress could be deployed, which stopped all user application deployments. The failures started right after an upgrade of the ModSec ingress controller module that had appeared to deploy successfully.
Symptom Deploys failing
Root cause Breaking change in an upgrade
A user reported the deployment error in the #ask-cloud-platform Slack channel at 13:15.
–
IL-0098 After 2i2c moved from ingress-nginx to nginx-ingress, a hub's Ingress took priority over cert-manager's temporary Ingress for the same host, so new hubs could not get TLS certificates2i2c · 2026same root cause · shared Ingress controller
IL-0092 Upgrading ingress-nginx from 1.12.0 to 1.14.3 brought a newer ModSecurity and OWASP Core Rule Set whose false positives blocked HMPPS Auth, so users could not sign in to their applicationsUK Ministry of Justice · 2026same root cause · shared Ingress controller
IL-0117 A scheduled maintenance and an ingress upgrade on OpenAI's user-facing clusters caused a networking control plane regression that briefly broke the data plane, and the services that restarted then overwhelmed a backend persistence store with a heavy first-start queryOpenAI · 2024same root cause · shared Ingress controller
IL-0203 A routine Kubernetes cluster system update made the AWS Load Balancer Controller fail, so new pods were never registered with the ALB and Honeycomb's EU1 ingest and UI went down once the last old pods were replacedHoneycomb · 2024same root cause · shared Ingress controller
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.
same root cause · shared Admission webhook
IL-0060 Kubernetes 1.24 upgrade removed the master node label that Calico route reflectors selected on, dropping all pod routes and taking Reddit down for 314 minutesReddit · 2023same root cause · shared Admission webhook