Incident Library

IL-0405 PortSwigger · 2026-01-11kubernetesawsnetworking

An expired AWS Load Balancer Controller webhook certificate left no healthy targets and took BSEE SaaS down

On 28 November 2025 PortSwigger upgraded the AWS Load Balancer Controller in its clusters from v1.14.1 to v1.16.0. A change to the controller's Helm chart in v1.15.0 (pull request 4359) changed how certificates are managed, and after the upgrade the old TLS secret was orphaned: it was owned by a cert-manager Certificate object that no longer existed, so cert-manager could not renew it.

The certificate in that secret expired on 8 January 2026. Webhook validation then failed and the controller stopped working. The webhook certificate is needed for load balancer target group configuration, so the controller could not update the AWS Target Groups, which eventually had no healthy targets, and from 10:00 UTC on 11 January all BSEE SaaS instances in the US and Canada regions were down. Customers reported it at 09:52 UTC on 12 January and the platform team began investigating at 10:00. The team deleted the orphaned aws-load-balancer-tls secrets in the US and Canada clusters, redeployed the controller configuration with Helm and checked that new certificates were generated and the webhooks worked. Services were restored at 10:40 UTC, about 24 hours 40 minutes after impact began. Two support cases were raised and no customer data was lost.

Severity
–
Scope
All BSEE SaaS instances in the US and Canada regions: customers could not access the Web UI or start scans
Users affected
US and Canada region customers; 2 support cases raised
Region
US and Canada regions
Time to detect
23h 52m
Time to resolve
24h 40m

Symptoms

From 10:00 UTC on 11 January 2026 every BSEE SaaS instance in PortSwigger's US and Canada regions was unavailable: customers could not open the Web UI or start scans. The platform team learned of it only when customers reported it through support at 09:52 UTC the next day.

Behind the outage, webhook validation for the AWS Load Balancer Controller failed because the webhook's TLS certificate had expired on 8 January 2026, three days before services stopped. Without a working webhook the controller could not update the AWS Target Group configurations, and the target groups eventually had zero healthy targets. When the platform team investigated, it found expired webhook certificates across all the affected clusters.

Symptom Service outageTLS certificate errors

Root cause

Root cause Breaking change in an upgrade

Component Ingress controllerAdmission webhookcert-managerHelmConfigMaps and SecretsCloud load balancer
Contributing factor Change with a delayed effectMonitoring blind spot
Trigger
The upgrade of the AWS Load Balancer Controller from v1.14.1 to v1.16.0 on 28 November 2025 left an orphaned webhook TLS secret that cert-manager could not renew, and the certificate in it expired on 8 January 2026.

Detection

Customers reported the outage through support at 09:52 UTC on 12 January 2026, almost 24 hours after impact began, and the platform team began investigating at 10:00 UTC.

Remediation

Action items

Lessons

–

Sources

References

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Ingress controller, Admission webhook, cert-manager among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.