Incident Library
Applied filters Root causeBreaking change in an upgradeComponentIngress controllerClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases5 cases

1 to 5 of 5

  1. IL-0098 After 2i2c moved from ingress-nginx to nginx-ingress, a hub's Ingress took priority over cert-manager's temporary Ingress for the same host, so new hubs could not get TLS certificates
    2i2c2026
    Symptoms

    An attempt to launch a new hub failed because its ingress could not obtain a new TLS certificate. The day before, the same engineer had recreated the support deployment on the 2i2c-aws-us cluster and seen its TLS certificate orders stuck pending.

    Misrouted requestsDeploys failing
    Trigger

    Creating new hub deployments that needed new TLS certificates on clusters recently migrated from ingress-nginx to nginx-ingress.

    Root cause
    Breaking change in an upgrade
    Component
    Ingress controllercert-manager
    Contributing factor
    Change with a delayed effect
    Detection

    An attempt to launch a new hub failed; the engineer had seen certificate orders stuck pending on another cluster the day before and declared an incident.

    Remediation
    • annotated the Ingress objects that need TLS so that cert-manager could edit them in place to complete the ACME challenge
    Action items
    • Follow-up tracked in https://github.com/2i2c-org/infrastructure/issues/8016
    • Follow-up tracked in https://github.com/2i2c-org/infrastructure/issues/3278
    Lessons

    Swapping one ingress controller for another can change how Ingress objects that share a host are combined. cert-manager's temporary challenge Ingress depends on that behaviour, so the breakage shows up when a deployment needs a new certificate. 2i2c notes that it got lucky: it had already learned of this possible cause while testing on an internal cluster, and changing the cert-manager configuration was quick.

    View case
  2. IL-0092 Upgrading ingress-nginx from 1.12.0 to 1.14.3 brought a newer ModSecurity and OWASP Core Rule Set whose false positives blocked HMPPS Auth, so users could not sign in to their applications
    UK Ministry of Justice2026
    Symptoms

    At 08:56 on 30 March 2026, about an hour and three quarters after the rollout of a new ingress-nginx to production began, the MoJ Cloud Platform team noticed messages in the #hmpps-auth-audit-registers Slack channel about ModSecurity unexpectedly blocking requests. Users in HMPPS also raised incidents with the MoJ service desk.

    High error rateService outage
    Trigger

    The Cloud Platform team began deploying ingress-nginx 1.14.3, on images maintained by Chainguard, to the three production ingresses at 07:11 on 30 March 2026.

    Root cause
    Breaking change in an upgrade
    Component
    Ingress controllerWeb application firewall rules
    Contributing factor
    Unrepresentative staging
    Detection

    At 08:56 the Cloud Platform team saw messages in the #hmpps-auth-audit-registers Slack channel showing ModSecurity blocking requests unexpectedly; HMPPS users also raised incidents with the MoJ service desk.

    Remediation
    • rolled the ingress-nginx deployment back to the previous version, from 09:14 to 09:43
    • HMPPS applied ModSecurity rule exclusions for the rules blocking traffic to HMPPS Auth
    • added a temporary global exclusion for the CRS rule triggered by the PostHog cookie
    • redeployed ingress-nginx 1.14.3 with both exclusions in place, completed at 14:00
    Action items
    • Rescope the PostHog cookie so it is no longer sent with requests to all *.justice.gov.uk services
    • Remove the temporary global exclusion once the PostHog cookie has expired from enough users' browsers
    • Remind service teams to run dev and test workloads through the non-production ingress controllers so future ModSecurity and CRS updates are tested
    • Run production and non-production workloads in separate clusters in Cloud Platform 3.0, so non-production workloads must use the non-production ingress
    Lessons

    An ingress controller upgrade that bundles a web application firewall also changes which requests are allowed, not only how they are routed. A beta or non-production ingress only catches false positives for traffic that actually passes through it, and teams that keep their test workloads on the production ingress miss them. A fix on the client side, such as rescoping a cookie, takes effect only when old cookies have expired in users' browsers, so the rule change has to wait for it or ship with an exclusion.

    View case
  3. IL-0117 A scheduled maintenance and an ingress upgrade on OpenAI's user-facing clusters caused a networking control plane regression that briefly broke the data plane, and the services that restarted then overwhelmed a backend persistence store with a heavy first-start query
    OpenAI2024
    Symptoms

    On August 16, 2024 from 11:38 AM to 1:16 PM PT, the OpenAI primary API was unreliable: ChatGPT conversations succeeded less often, and logging in and creating accounts were affected. The status page called it elevated error rates for ChatGPT and the Platform API. The incident came in two waves, of 44 and 15 minutes.

    High error rateDatabase overload
    Trigger

    A scheduled maintenance combined with an upgrade to the ingress of OpenAI's user-facing clusters.

    Root cause
    Breaking change in an upgrade
    Component
    Ingress controllerApplication database
    Contributing factor
    No staged rolloutAutomation amplification
    Action items
    • Mitigate the networking control plane regression and validate that control plane restarts do not interfere with the clusters' data plane
    • Change service software to start faster and remove the first-start query, relieving the persistence layer
    • Deploy configuration changes that optimise the networking control plane's effect on the clusters' ability to handle traffic
    • Remove the expensive database query from critical startup paths
    • Add monitoring and alerting for networking control plane issues
    • Introduce staged rollouts with longer soak time for infrastructure changes
    • Audit systems for other slow queries that may affect service start time
    Lessons

    A brief connectivity loss turned into a long outage because recovery itself was expensive: the services that restarted overwhelmed one backend persistence store with a heavy first-start query. Keeping costly queries off startup paths shortens recovery, and staged rollouts with longer soak time limit how many systems an infrastructure regression reaches before it is caught.

    View case
  4. IL-0203 A routine Kubernetes cluster system update made the AWS Load Balancer Controller fail, so new pods were never registered with the ALB and Honeycomb's EU1 ingest and UI went down once the last old pods were replaced
    Honeycomb2024
    Symptoms

    Event ingestion and the Honeycomb UI in EU1 became unavailable at 22:00:10 UTC on April 9, 2024, and the status page reported that the EU1 instance was down. Both the ingestion service and the service behind the web UI stopped at roughly the same moment.

    Service outageStale endpoint routing
    Trigger

    A routine system update to the Kubernetes cluster, rolled out earlier on April 9, followed by a regular deployment that replaced pods one at a time.

    Root cause
    Breaking change in an upgrade
    Component
    Ingress controllerCloud load balancer
    Contributing factor
    Monitoring blind spotUnrepresentative stagingChange with a delayed effect
    Detection

    The first status post, saying the EU1 instance was down, went up at 15:08 PDT (22:08 UTC), about 8 minutes after the outage began. There was no alerting on the AWS Load Balancer Controller.

    Remediation
    • Restarted the AWS Load Balancer Controller
    • Rolled back the Kubernetes cluster system update
    Action items
    • Find the exact cause of the failure so the system update can be deployed safely
    • Add monitoring and alerting on the AWS Load Balancer Controller so on-call engineers learn of a failure quickly and can pause deployments
    Lessons

    A rolling deployment can look healthy while the component that wires new pods into the load balancer is broken; the failure only shows when the last old pod goes away. Honeycomb now treats the AWS Load Balancer Controller as a critical component that needs its own alerting, so deployments can be paused before they remove the last pods that still receive traffic. Extensive testing in development and staging did not reveal the bad interaction.

    View case
  5. IL-0090 After an upgrade of the ModSec ingress controller module, its validating webhook served a certificate for the wrong service name and blocked every Ingress create and change
    UK Ministry of Justice2021
    Symptoms

    At 13:15 on 9 June 2021 a user reported in the #ask-cloud-platform channel that deploying a UAT application failed on its Ingress resource.

    Deploys failing
    Trigger

    The Cloud Platform team upgraded the ModSec ingress controller module to v3.33.0; the deployment appeared to succeed.

    Root cause
    Breaking change in an upgrade
    Component
    Admission webhookIngress controller
    Contributing factor
    Unrepresentative staging
    Detection

    A user reported the deployment error in the #ask-cloud-platform Slack channel at 13:15.

    Remediation
    • rolled back the ModSec ingress controller module to v0.0.7
    Action items
    • Find out why the issue was not flagged in the test cluster and try to reproduce it, possibly with a new test (ticket #2972)
    • Add a smoke test that checks the alerts in Alertmanager (ticket #2973)
    • Add a helloworld app that uses the ModSec controller so smoke tests can check that traffic works (ticket #2974)
    • Make a new version of the ModSec module work on EKS for both live-1 and live; neither the old nor the new version works on live (ticket #2975)
    View case
Check your cluster for this

These 5 cases involve Ingress controller and Admission webhook. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Breaking change in an upgrade5 cases

Component
Ingress controller5Admission webhook1cert-manager1Cloud load balancer1Application database1Web application firewall rules1
Contributing factor
Unrepresentative staging3Change with a delayed effect2Automation amplification1Monitoring blind spot1No staged rollout1
Organisation
UK Ministry of Justice22i2c1Honeycomb1OpenAI1
  1. IL-0098 After 2i2c moved from ingress-nginx to nginx-ingress, a hub's Ingress took priority over cert-manager's temporary Ingress for the same host, so new hubs could not get TLS certificates
    2i2c2026
  2. IL-0092 Upgrading ingress-nginx from 1.12.0 to 1.14.3 brought a newer ModSecurity and OWASP Core Rule Set whose false positives blocked HMPPS Auth, so users could not sign in to their applications
    UK Ministry of Justice2026
  3. IL-0117 A scheduled maintenance and an ingress upgrade on OpenAI's user-facing clusters caused a networking control plane regression that briefly broke the data plane, and the services that restarted then overwhelmed a backend persistence store with a heavy first-start query
    OpenAI2024
  4. IL-0203 A routine Kubernetes cluster system update made the AWS Load Balancer Controller fail, so new pods were never registered with the ALB and Honeycomb's EU1 ingest and UI went down once the last old pods were replaced
    Honeycomb2024
  5. IL-0090 After an upgrade of the ModSec ingress controller module, its validating webhook served a certificate for the wrong service name and blocked every Ingress create and change
    UK Ministry of Justice2021