Cases23 cases

3 to 7 of 23

1. IL-0405 An expired AWS Load Balancer Controller webhook certificate left no healthy targets and took BSEE SaaS down

   PortSwigger2026

   Symptoms

   From 10:00 UTC on 11 January 2026 every BSEE SaaS instance in PortSwigger's US and Canada regions was unavailable: customers could not open the Web UI or start scans. The platform team learned of it only when customers reported it through support at 09:52 UTC the next day.

   Service outageTLS certificate errors

   Trigger

   The upgrade of the AWS Load Balancer Controller from v1.14.1 to v1.16.0 on 28 November 2025 left an orphaned webhook TLS secret that cert-manager could not renew, and the certificate in it expired on 8 January 2026.

   Root cause

   Breaking change in an upgrade

   Component

   Ingress controllerAdmission webhookcert-managerHelmConfigMaps and SecretsCloud load balancer

   Contributing factor

   Change with a delayed effectMonitoring blind spot

   Detection

   Customers reported the outage through support at 09:52 UTC on 12 January 2026, almost 24 hours after impact began, and the platform team began investigating at 10:00 UTC.

   Remediation

   • Deleted the orphaned aws-load-balancer-tls secrets in the US and Canada clusters
   • Redeployed the AWS Load Balancer Controller configuration with Helm
   • Verified that new certificates were generated and the webhooks worked again

   Action items

   • Audit all clusters for orphaned secrets that have no owning Certificate resource and resolve them
   • Add high-level application access alerting for each SaaS cluster from an external probe

   View case
2. IL-0188 Rolling updates on EKS returned 502 errors for about 30 seconds because Kubernetes deleted old pods as soon as new pods passed readiness, before the ALB had marked the new pod targets healthy

   Yonghyun Kim2025

   Symptoms

   Every rolling update of the team's microservices on Amazon EKS, behind an Application Load Balancer, was followed by intermittent 502 and 503 errors for users. A Slack message at 3 a.m. read "배포는 성공했는데, 서비스는 죽었다" (the deployment succeeded, but the service died). Every pod was Ready and Datadog was green while users saw errors.

   High error rateService outage

   Trigger

   A rolling update of a Deployment served through an ALB that registers pod IPs as targets, with no pod readiness gates.

   Root cause

   Faulty deployment

   Component

   Cloud load balancerIngress controllerPod termination

   Contributing factor

   Unverified assumptionUnreviewed defaultsMonitoring blind spot

   Detection

   A Slack message at 3 a.m. reported that the deployment had succeeded but the service was down; Datadog dashboards were green at the time.

   Remediation

   • labelled the namespace with elbv2.k8s.aws/pod-readiness-gate-inject=enabled so the AWS Load Balancer Controller injects pod readiness gates and Kubernetes waits for the ALB target to be healthy before deleting old pods

   Lessons

   A pod being Ready in Kubernetes does not mean the load balancer will send it traffic. Without a readiness gate, a rolling update can delete the old pods while the ALB still considers the new targets unhealthy, and dashboards that track pod status stay green during the errors. Making pods wait longer on shutdown reduced the errors but did not fix the start-up gap and made every deployment much slower; the readiness gate ties the pod's Ready state to the ALB's view of the target.

   View case
3. IL-0150 A latent race condition in DynamoDB's DNS automation left an empty DNS record for dynamodb.us-east-1.amazonaws.com, and the DynamoDB and EC2 launch failures that followed caused EKS, ECS and Fargate container launch failures and cluster scaling delays in us-east-1

   Amazon Web Services2025

   Symptoms

   At 11:48 PM PDT on 19 October 2025, every system that reached DynamoDB in us-east-1 through the public regional endpoint started getting DNS failures and could not connect. This included customer traffic and internal AWS services that depend on DynamoDB. DynamoDB global table replicas in other regions still answered but fell behind in replication to and from us-east-1.

   DNS resolution failuresHigh error rateScale-up failureIntermittent connection failuresHigh latency

   Trigger

   One DNS Enactor, slowed by unusually high retry delays, applied a much older DNS plan to the regional DynamoDB endpoint just as another Enactor that had applied the newest plan ran its clean-up, which deleted that older plan and with it every IP address of the endpoint.

   Root cause

   Software defect

   Component

   Authoritative DNS servers and recordsDatabaseCloud load balancer

   Contributing factor

   Automation amplificationRetry stormNo runbook

   Detection

   DNS failures to DynamoDB began immediately at 11:48 PM PDT and the engineering teams of affected AWS services were engaged at once; by 12:38 AM they had identified DynamoDB's DNS state as the source. The later NLB health check problem was detected by monitoring at 6:52 AM.

   Remediation

   • applied temporary mitigations by 1:15 AM that let some internal services reach DynamoDB and repaired key internal tooling
   • restored all DynamoDB DNS information by 2:25 AM through manual operator intervention
   • throttled incoming work and selectively restarted DWFM hosts from 4:14 AM so droplet leases could be re-established
   • reduced the load on Network Manager to bring network configuration propagation times back to normal by 10:36 AM
   • disabled automatic health check failover for NLB at 9:36 AM, and re-enabled it at 2:09 PM after EC2 recovered
   • relaxed the EC2 request throttles from 11:23 AM until all EC2 APIs and launches were normal at 1:50 PM

   Action items

   • Disable the DynamoDB DNS Planner and DNS Enactor automation worldwide
   • Fix the race condition and add protections against applying incorrect DNS plans before re-enabling the DNS automation
   • Add a velocity control to NLB that limits how much capacity a single load balancer can remove when health check failures cause availability zone failover
   • Build an additional EC2 test suite that exercises the DWFM recovery workflow at scale
   • Improve throttling in EC2 data propagation systems so incoming work is rate limited by the size of the waiting queue

   Lessons

   A single DNS record that automation can empty is enough to take a widely used regional endpoint off the network, and every service built on it fails in turn. A check made once at the start of a long operation can be stale by the time the operation writes its result. Recovery of the dependency is not recovery of the dependents: systems that lost state during the outage, such as EC2's droplet leases and network propagation backlog, can collapse under their own catch-up work, and container services that rely on new instance launches keep failing to launch and scale until that work is done.

   View case
4. IL-0079 A GKE node auto-upgrade to an ubuntu_containerd image with a kernel regression left kube-proxy unable to apply its ip6tables rules, so load balancer health checks failed and several clusters lost all ingress traffic

   Altinity2025

   Symptoms

   On a Saturday, an OpsGenie alert fired for a ClickHouse cluster in Altinity.Cloud that had gone silent: no queries, no metrics, and no logs from customers trying to connect. The pods were running and the nodes looked healthy, but no traffic reached the database. Other clusters soon showed the same thing, and external access was blocked across several GKE clusters.

   Service outage

   Trigger

   GKE automatically upgraded worker nodes to the ubuntu_containerd image 1.32.3-gke.1785003, which ships kernel 6.8.0-1022-gke.

   Root cause

   Breaking change in an upgrade

   Component

   kube-proxyNode operating system imageLinux kernelCloud load balancer

   Contributing factor

   Unpatched known bugNo staged rollout

   Detection

   An OpsGenie alert fired when one ClickHouse cluster stopped receiving queries, metrics and customer connection logs.

   Remediation

   • added maintenance exceptions to pause all upgrades on the GKE clusters that were not yet affected
   • changed the load balancer health check policy to ignore failed kube-proxy health checks, so traffic flowed to the nodes again
   • began migrating the affected clusters from the ubuntu_containerd to the cos_containerd node image by recreating nodes

   Action items

   • Expand image pre-testing pipelines to catch regressions earlier
   • Enable notifications for upcoming GKE auto-upgrades
   • Make GKE maintenance windows required across the fleet to control when upgrades roll out
   • Track Google's issue and wait for a patched Ubuntu image

   Lessons

   A managed node upgrade can break kube-proxy in a way that leaves pods and nodes looking healthy while the load balancer sends nothing, because the load balancer health check depends on kube-proxy. kube-proxy's own logs showed the cause. Altinity recommends that GKE users on ubuntu_containerd restore traffic by switching to externalTrafficPolicy: Cluster, which disables the kube-proxy health check in the load balancer, move production workloads to cos_containerd, and pause auto-upgrades where possible until a patched image has been confirmed and tested.

   View case
5. IL-0542 Twingate authentication and authorization requests failed at higher rates after traffic moved onto a smaller GKE cluster

   Twingate2025

   Symptoms

   On 21 March 2025, between 21:10 and 21:41 UTC, part of the authentication and authorization requests to Twingate services failed with elevated error rates.

   High error rate

   Trigger

   Twingate saw anomalies in one of its GKE clusters and scaled down the deployments in that cluster to protect the rest of the system, which moved its traffic to the other clusters.

   Root cause

   Cloud provider outage

   Component

   Cloud load balancerGKE control planeHorizontal pod autoscalerCluster autoscaler

   Contributing factor

   Retry stormNo headroomOverlapping failuresHarmful manual action

   Detection

   Twingate observed anomalies in one of its GKE clusters at the start of the incident.

   Remediation

   • Reintroduced capacity in the cluster that had been scaled down, so traffic was rebalanced across regions

   Action items

   • Keep working with the cloud provider to find the cause of the unexpected timeouts and of the cluster update's impact
   • Run deployments in all regions with the same number of replicas in the HPA configuration
   • Raise the maximum node counts of the cluster node autoscalers to leave more room to scale up
   • Configure the HPA to scale up faster

   View case

By root cause7