Cases2 cases

1 to 2 of 2

1. IL-0405 An expired AWS Load Balancer Controller webhook certificate left no healthy targets and took BSEE SaaS down

   PortSwigger2026

   Symptoms

   From 10:00 UTC on 11 January 2026 every BSEE SaaS instance in PortSwigger's US and Canada regions was unavailable: customers could not open the Web UI or start scans. The platform team learned of it only when customers reported it through support at 09:52 UTC the next day.

   Service outageTLS certificate errors

   Trigger

   The upgrade of the AWS Load Balancer Controller from v1.14.1 to v1.16.0 on 28 November 2025 left an orphaned webhook TLS secret that cert-manager could not renew, and the certificate in it expired on 8 January 2026.

   Root cause

   Breaking change in an upgrade

   Component

   Ingress controllerAdmission webhookcert-managerHelmConfigMaps and SecretsCloud load balancer

   Contributing factor

   Change with a delayed effectMonitoring blind spot

   Detection

   Customers reported the outage through support at 09:52 UTC on 12 January 2026, almost 24 hours after impact began, and the platform team began investigating at 10:00 UTC.

   Remediation

   • Deleted the orphaned aws-load-balancer-tls secrets in the US and Canada clusters
   • Redeployed the AWS Load Balancer Controller configuration with Helm
   • Verified that new certificates were generated and the webhooks worked again

   Action items

   • Audit all clusters for orphaned secrets that have no owning Certificate resource and resolve them
   • Add high-level application access alerting for each SaaS cluster from an external probe

   View case
2. IL-0032 Mutating admission controller generated broken JSON patches, silently blocking deploys for a week until a production service ran down to two pods

   Airbnb2020

   Symptoms

   A production service at Airbnb showed a high error rate and could not be deployed. The deploy failed with a cryptic Kubernetes error about being unable to access invalid index 5.

   High error rateThroughput degradationDeploys failing

   Trigger

   A change to the mutating admission controller, which had been rolled out within the previous month, had gone in seven days before the incident surfaced; when it had to delete two items from an array in a spec, it generated an invalid JSON patch that removed index 2 and then index 3.

   Root cause

   Software defect

   Component

   Admission webhookHorizontal pod autoscaler

   Contributing factor

   Change with a delayed effectMonitoring blind spotIgnored early signal

   Detection

   The affected service reported a high error rate and its deploy failed; the slow degradation of other services over the previous seven days had gone unnoticed.

   Remediation

   • deleted the mutating admission controller from the affected cluster

   Action items

   • Create alerts from the existing metrics that would have shown the problem earlier

   Lessons

   Keep in mind that a mutating admission controller exists in the cluster; a recently added one is easy to forget during an incident. Search for the exact error message early instead of working on the visible symptoms. Page a teammate when progress stalls.

   View case

By root cause2