Incident Library
Applied filters ComponentAdmission webhookFactorChange with a delayed effectDomainKubernetesClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0405 An expired AWS Load Balancer Controller webhook certificate left no healthy targets and took BSEE SaaS down
    PortSwigger2026
    Symptoms

    From 10:00 UTC on 11 January 2026 every BSEE SaaS instance in PortSwigger's US and Canada regions was unavailable: customers could not open the Web UI or start scans. The platform team learned of it only when customers reported it through support at 09:52 UTC the next day.

    Service outageTLS certificate errors
    Trigger

    The upgrade of the AWS Load Balancer Controller from v1.14.1 to v1.16.0 on 28 November 2025 left an orphaned webhook TLS secret that cert-manager could not renew, and the certificate in it expired on 8 January 2026.

    Root cause
    Breaking change in an upgrade
    Component
    Ingress controllerAdmission webhookcert-managerHelmConfigMaps and SecretsCloud load balancer
    Contributing factor
    Change with a delayed effectMonitoring blind spot
    Detection

    Customers reported the outage through support at 09:52 UTC on 12 January 2026, almost 24 hours after impact began, and the platform team began investigating at 10:00 UTC.

    Remediation
    • Deleted the orphaned aws-load-balancer-tls secrets in the US and Canada clusters
    • Redeployed the AWS Load Balancer Controller configuration with Helm
    • Verified that new certificates were generated and the webhooks worked again
    Action items
    • Audit all clusters for orphaned secrets that have no owning Certificate resource and resolve them
    • Add high-level application access alerting for each SaaS cluster from an external probe
    View case
  2. IL-0032 Mutating admission controller generated broken JSON patches, silently blocking deploys for a week until a production service ran down to two pods
    Airbnb2020
    Symptoms

    A production service at Airbnb showed a high error rate and could not be deployed. The deploy failed with a cryptic Kubernetes error about being unable to access invalid index 5.

    High error rateThroughput degradationDeploys failing
    Trigger

    A change to the mutating admission controller, which had been rolled out within the previous month, had gone in seven days before the incident surfaced; when it had to delete two items from an array in a spec, it generated an invalid JSON patch that removed index 2 and then index 3.

    Root cause
    Software defect
    Component
    Admission webhookHorizontal pod autoscaler
    Contributing factor
    Change with a delayed effectMonitoring blind spotIgnored early signal
    Detection

    The affected service reported a high error rate and its deploy failed; the slow degradation of other services over the previous seven days had gone unnoticed.

    Remediation
    • deleted the mutating admission controller from the affected cluster
    Action items
    • Create alerts from the existing metrics that would have shown the problem earlier
    Lessons

    Keep in mind that a mutating admission controller exists in the cluster; a recently added one is easy to forget during an incident. Search for the exact error message early instead of working on the visible symptoms. Page a teammate when progress stalls.

    View case
Check your cluster for this

These 2 cases involve Admission webhook and cert-manager. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause2

Breaking change in an upgrade1 case

Component
Admission webhookcert-managerCloud load balancerConfigMaps and SecretsHelmIngress controller
Contributing factor
Change with a delayed effectMonitoring blind spot
Organisation
PortSwigger
  1. IL-0405 An expired AWS Load Balancer Controller webhook certificate left no healthy targets and took BSEE SaaS down
    PortSwigger2026

Software defect1 case

Component
Admission webhookHorizontal pod autoscaler
Contributing factor
Change with a delayed effectIgnored early signalMonitoring blind spot
Organisation
Airbnb
  1. IL-0032 Mutating admission controller generated broken JSON patches, silently blocking deploys for a week until a production service ran down to two pods
    Airbnb2020