Cases6 cases

1 to 5 of 6

1. IL-0124 Capacity changes queued while a circuit breaker was engaged ran all at once when it was released, taking multiple OpenAI inference engines out of service and raising API error rates

   OpenAI2026

   Symptoms

   Between about 4:26 PM and 4:56 PM PST on 4 March 2026, some OpenAI API requests to several models failed or took longer than usual, and some Codex traffic was affected too. The overall API success rate dropped temporarily.

   High error rateHigh latency

   Trigger

   A protective circuit breaker that had paused some scheduling operations was released, and the infrastructure actions queued during the pause were executed at the same time.

   Root cause

   Software defect

   Component

   Automated capacity manager

   Contributing factor

   Automation amplificationChange with a delayed effectNo safeguard on destructive changeMonitoring blind spot

   Remediation

   • manually reassigned infrastructure resources to the affected models
   • brought the removed inference engines back online and reassigned them until error rates returned to normal

   Action items

   • Improve safeguards in the automated capacity management systems so they detect and respond to state drift
   • Improve observability of circuit breakers and scheduling systems so their status and effects are easier to identify
   • Add safeguards so queued actions cannot run in large batches after long pauses
   • Improve monitoring and alerting around capacity reconciliation problems

   Lessons

   Pausing one automated system does not stop the systems that feed it. Changes queued during the pause reflect the state at the time they were queued, and releasing the queue all at once applies those stale decisions together. The fixes OpenAI names are detecting state drift, making circuit breaker status visible, and preventing large batches from executing after a long pause.

   View case
2. IL-0150 A latent race condition in DynamoDB's DNS automation left an empty DNS record for dynamodb.us-east-1.amazonaws.com, and the DynamoDB and EC2 launch failures that followed caused EKS, ECS and Fargate container launch failures and cluster scaling delays in us-east-1

   Amazon Web Services2025

   Symptoms

   At 11:48 PM PDT on 19 October 2025, every system that reached DynamoDB in us-east-1 through the public regional endpoint started getting DNS failures and could not connect. This included customer traffic and internal AWS services that depend on DynamoDB. DynamoDB global table replicas in other regions still answered but fell behind in replication to and from us-east-1.

   DNS resolution failuresHigh error rateScale-up failureIntermittent connection failuresHigh latency

   Trigger

   One DNS Enactor, slowed by unusually high retry delays, applied a much older DNS plan to the regional DynamoDB endpoint just as another Enactor that had applied the newest plan ran its clean-up, which deleted that older plan and with it every IP address of the endpoint.

   Root cause

   Software defect

   Component

   Authoritative DNS servers and recordsApplication databaseCloud load balancer

   Contributing factor

   Automation amplificationRetry stormNo runbook

   Detection

   DNS failures to DynamoDB began immediately at 11:48 PM PDT and the engineering teams of affected AWS services were engaged at once; by 12:38 AM they had identified DynamoDB's DNS state as the source. The later NLB health check problem was detected by monitoring at 6:52 AM.

   Remediation

   • applied temporary mitigations by 1:15 AM that let some internal services reach DynamoDB and repaired key internal tooling
   • restored all DynamoDB DNS information by 2:25 AM through manual operator intervention
   • throttled incoming work and selectively restarted DWFM hosts from 4:14 AM so droplet leases could be re-established
   • reduced the load on Network Manager to bring network configuration propagation times back to normal by 10:36 AM
   • disabled automatic health check failover for NLB at 9:36 AM, and re-enabled it at 2:09 PM after EC2 recovered
   • relaxed the EC2 request throttles from 11:23 AM until all EC2 APIs and launches were normal at 1:50 PM

   Action items

   • Disable the DynamoDB DNS Planner and DNS Enactor automation worldwide
   • Fix the race condition and add protections against applying incorrect DNS plans before re-enabling the DNS automation
   • Add a velocity control to NLB that limits how much capacity a single load balancer can remove when health check failures cause availability zone failover
   • Build an additional EC2 test suite that exercises the DWFM recovery workflow at scale
   • Improve throttling in EC2 data propagation systems so incoming work is rate limited by the size of the waiting queue

   Lessons

   A single DNS record that automation can empty is enough to take a widely used regional endpoint off the network, and every service built on it fails in turn. A check made once at the start of a long operation can be stale by the time the operation writes its result. Recovery of the dependency is not recovery of the dependents: systems that lost state during the outage, such as EC2's droplet leases and network propagation backlog, can collapse under their own catch-up work, and container services that rely on new instance launches keep failing to launch and scale until that work is done.

   View case
3. IL-0171 Karpenter node auto-repair replaced EKS GPU nodes about 10 minutes after they became Ready, in a loop, because the Bottlerocket NVIDIA AMI lacked libdcgm.so and set AcceleratedHardwareReady to False

   Inflab2025

   Symptoms

   GPU nodes for an AI video upscaling job were created normally and reached Ready, and then about 10 to 11 minutes later Karpenter treated each one as unhealthy and replaced it. Every new node went the same way, so the GPU pods kept restarting and the upscaling work could not make progress.

   Node recreation loopGPU unavailable

   Trigger

   Enabling Karpenter's node auto-repair feature gate (nodeRepair: true) and then running GPU pods on nodes from the Bottlerocket NVIDIA AMI.

   Root cause

   Software defect

   Component

   KarpenterNode operating system imageGPU driver

   Contributing factor

   Automation amplificationUnverified assumption

   Detection

   The team noticed GPU nodes being replaced repeatedly after running GPU pods, then found the DCGMError message in the node events.

   Remediation

   • Disabled Karpenter node auto-repair by setting the nodeRepair feature gate to false

   Lessons

   Automatic node repair acts on every unhealthy node condition, including AcceleratedHardwareReady, so an image defect that only breaks GPU health monitoring is enough to have every GPU node terminated and recreated in a loop. Before turning on auto-repair, check which conditions it watches and whether the node image reports them correctly. Disabling auto-repair stops the loop at once but is only a temporary measure; installing the missing library or changing the AMI address the cause but cost more on an immutable OS such as Bottlerocket.

   View case
4. IL-0147 AKS nodes behind an HTTP proxy failed to provision with CSE exit code 31 on 202508 node images, because a refactoring stopped the proxy from being used when downloading uncached Kubernetes binaries

   Microsoft2025

   Symptoms

   Creating or scaling a node pool on AKS 1.31.8 failed: the VMSS instances never finished provisioning. This started after planned maintenance moved the node image from 202507.* to 202508.*, in a cluster where nodes must send all outbound traffic through an internal HTTP/HTTPS proxy and direct egress is blocked. It happened with both Ubuntu and AzureLinux images.

   Scale-up failureUpgrade stall

   Trigger

   A node image auto-upgrade from 202507.* to 202508.* during planned maintenance, on clusters whose Kubernetes patch version was not pre-cached in the new image.

   Root cause

   Software defect

   Component

   Node operating system imageNode pool

   Contributing factor

   Cache-masked failureAutomation amplification

   Detection

   The customer saw VMSS instance provisioning fail with a vmssCSE error in the VMSS activity logs and reported it as a GitHub issue.

   Remediation

   • The customer pinned the node image to a 202507.* snapshot on clusters running 1.31.8
   • The customer upgraded clusters to AKS 1.31.9 or 1.31.10, whose binaries are cached in the 202508.* images
   • The AKS team hot-patched VHD 20250811 so the proxy configuration is used for uncached binary downloads and made it globally available in public regions on 2025-09-04; it also announced the hotfix for VHD 20250820
   • The AKS team told users to reimage their nodes with az aks nodepool upgrade --node-image-only to pick up the hotfixed image

   Action items

   • Apply the proxy fix in all future VHDs

   Lessons

   A bootstrap path that runs only when an artefact is missing from a cache is easy to break without noticing: here only clusters on Kubernetes versions not pre-cached in the node image ever downloaded binaries, so only they met the proxy regression. Testing node provisioning behind a mandatory egress proxy, on older patch versions as well as the newest, would exercise that path. For clusters with restricted egress, pinning the node image or moving to a Kubernetes version cached in the image were both quick workarounds while the fix was shipped.

   View case
5. IL-0112 The daily node rollover put DFDS's EKS nodes on an AMI later recalled for a kernel issue, causing "PLEG is not healthy" NodeNotReady flaps and pod network timeouts

   DFDS2022

   Symptoms

   On 2022-10-31 a planned upgrade of the vpc-cni EKS add-on, started from the deployment pipeline at 13:45 UTC, timed out at 14:10. Only about 10 of the 45 aws-node pods had upgraded, 5 were stuck in Terminating and the other 30 had not been attempted.

   Node NotReadyUpgrade stallIntermittent timeoutsIntermittent connection failures

   Trigger

   At 07:30 UTC on 2022-10-31 the daily node rollover replaced the existing nodes with nodes built from AMI v20221027, the latest AWS EKS node AMI, released on 2022-10-29 and later recalled because of a kernel issue.

   Root cause

   Software defect

   Component

   Node operating system imageLinux kernelkubeletAWS VPC CNINode launch templateTerraform

   Contributing factor

   No staged rolloutAutomation amplificationOverlapping failuresMonitoring blind spotNo health probes

   Detection

   The team noticed nodes flapping between NodeNotReady and NodeReady ("KubeletNotReady PLEG is not healthy") at 14:30 UTC while watching the stalled vpc-cni upgrade, about seven hours after the rollover. Time to detection was 7 hours. Alerting did not catch the NodeNotReady events or the socket growth.

   Remediation

   • cordoned, drained and deleted 15 stuck nodes one at a time to complete the vpc-cni upgrade
   • changed the Terraform infrastructure-modules so a specific AMI could be set
   • rolled the EC2 Auto Scaling group launch configuration back to the previous known good AMI through the pipeline

   Action items

   • Reach out to teams with suggestions about readiness and liveness probes
   • Create a dashboard for readiness and liveness probe coverage
   • Write a wiki article on readiness and liveness probes and their usefulness
   • Modify infrastructure-modules to allow specifying a particular AMI
   • Schedule a discussion on whether to take the latest AMI immediately and how to test new AMIs before they are deployed
   • Implement alerting on abnormal NodeNotReady events and on increased socket use
   • Create a learning path for Kubernetes within the team
   • Collect logs from nodes centrally
   • Create a Prometheus exporter that retrieves the latest available AMI, with a dashboard and alerts

   Lessons

   When a new node AMI has just been rolled out, upgrading other cluster components on the same day makes it hard to tell which change is causing trouble; here the vpc-cni upgrade on the same day confused matters. Taking the latest AMI automatically with no way to pin one meant a rollback first needed a Terraform change. The team also concluded that it needs better communication paths with application teams during incidents.

   View case

By root cause1