Cases3 cases

1 to 3 of 3

1. IL-0150 A latent race condition in DynamoDB's DNS automation left an empty DNS record for dynamodb.us-east-1.amazonaws.com, and the DynamoDB and EC2 launch failures that followed caused EKS, ECS and Fargate container launch failures and cluster scaling delays in us-east-1

   Amazon Web Services2025

   Symptoms

   At 11:48 PM PDT on 19 October 2025, every system that reached DynamoDB in us-east-1 through the public regional endpoint started getting DNS failures and could not connect. This included customer traffic and internal AWS services that depend on DynamoDB. DynamoDB global table replicas in other regions still answered but fell behind in replication to and from us-east-1.

   DNS resolution failuresHigh error rateScale-up failureIntermittent connection failuresHigh latency

   Trigger

   One DNS Enactor, slowed by unusually high retry delays, applied a much older DNS plan to the regional DynamoDB endpoint just as another Enactor that had applied the newest plan ran its clean-up, which deleted that older plan and with it every IP address of the endpoint.

   Root cause

   Software defect

   Component

   Authoritative DNS servers and recordsDatabaseCloud load balancer

   Contributing factor

   Automation amplificationRetry stormNo runbook

   Detection

   DNS failures to DynamoDB began immediately at 11:48 PM PDT and the engineering teams of affected AWS services were engaged at once; by 12:38 AM they had identified DynamoDB's DNS state as the source. The later NLB health check problem was detected by monitoring at 6:52 AM.

   Remediation

   • applied temporary mitigations by 1:15 AM that let some internal services reach DynamoDB and repaired key internal tooling
   • restored all DynamoDB DNS information by 2:25 AM through manual operator intervention
   • throttled incoming work and selectively restarted DWFM hosts from 4:14 AM so droplet leases could be re-established
   • reduced the load on Network Manager to bring network configuration propagation times back to normal by 10:36 AM
   • disabled automatic health check failover for NLB at 9:36 AM, and re-enabled it at 2:09 PM after EC2 recovered
   • relaxed the EC2 request throttles from 11:23 AM until all EC2 APIs and launches were normal at 1:50 PM

   Action items

   • Disable the DynamoDB DNS Planner and DNS Enactor automation worldwide
   • Fix the race condition and add protections against applying incorrect DNS plans before re-enabling the DNS automation
   • Add a velocity control to NLB that limits how much capacity a single load balancer can remove when health check failures cause availability zone failover
   • Build an additional EC2 test suite that exercises the DWFM recovery workflow at scale
   • Improve throttling in EC2 data propagation systems so incoming work is rate limited by the size of the waiting queue

   Lessons

   A single DNS record that automation can empty is enough to take a widely used regional endpoint off the network, and every service built on it fails in turn. A check made once at the start of a long operation can be stale by the time the operation writes its result. Recovery of the dependency is not recovery of the dependents: systems that lost state during the outage, such as EC2's droplet leases and network propagation backlog, can collapse under their own catch-up work, and container services that rely on new instance launches keep failing to launch and scale until that work is done.

   View case
2. IL-0232 Calico pods in the codfw WikiKube cluster hit their resource limits and crash-looped, taking nodes out one by one

   Wikimedia Foundation2024

   Symptoms

   At 05:01 UTC on 9 October 2024 a page fired for GatewayBackendErrorsHigh, elevated 5xx errors from page-analytics_cluster in codfw, followed by a growing number of KubernetesCalicoDown alerts for WikiKube nodes not running the calico-node pod and by MediaWikiMemcachedHighErrorRate. For about an hour uncached MediaWiki requests, such as those of logged-in users, edits and API calls, saw high latency or errors.

   High error rateHigh latencyContainer crash loopProbe failuresControl plane lag

   Trigger

   An operator used the new mwscript-k8s utility to run a batch of video transcode maintenance, the TimedMediaHandler requeueTranscodes.php script. mwscript-k8s deploys a whole helm release for every script invocation, so the batch created close to 2k Calico NetworkPolicies along with many other resources over several hours.

   Root cause

   Capacity exhaustion

   Component

   CalicoHelmJobs and CronJobs

   Contributing factor

   Change with a delayed effectNo runbook

   Detection

   Automated monitoring detected the issue and paged the on-call SRE: GatewayBackendErrorsHigh at 05:01:36 UTC, 16 KubernetesCalicoDown alerts at 05:03:32 UTC and MediaWikiMemcachedHighErrorRate at 05:04:15 UTC.

   Remediation

   • Raised the resource limits of the Calico and Typha pods and applied the change with helmfile, which redeployed all Calico pods

   Action items

   • Stop mwscript-k8s from creating so many resources

   Lessons

   The incident came close to a full outage. Automated monitoring caught it and many responders found the cause, but end users were affected and responders were unsure whether they could switch over to eqiad. The problem was noticed just before the hand-off from Americas to EMEA SRE shifts, a period with a long-known lack of coverage, and it helped that the transcodes were not queued on a Friday evening in the US.

   View case
3. IL-0009 A duplicated generated name made Argo CD replace the InfluxDB Cloud production Application and delete the whole production environment

   InfluxData2022

   Symptoms

   Right after a pull request to the infrastructure-as-code repository was merged, every API endpoint of one InfluxDB Cloud production cluster began returning 404 errors. Writes and reads failed alike: no one could collect data, scheduled tasks failed to run and external queries did not work.

   Service outageResources deletedHigh error rate

   Trigger

   A merged pull request whose rendered YAML added a new Argo CD Application and AppProject that an automation error had named app01-cluster01, the name of the existing production Application, instead of app02-cluster01; reviewers did not notice the app01 and app02 difference.

   Root cause

   Configuration change

   Component

   Argo CDPersistent volumes and stateful setsConfiguration templating

   Contributing factor

   Missed in reviewNo runbookNo safeguard on destructive changeMonitoring blind spotAutomation amplification

   Detection

   Monitoring and alerting did not detect the issue right away, so the team's first response was to try to roll the change back rather than plan a recovery. The write-up does not say how the team found out.

   Remediation

   • stopped the revert of the merged pull request
   • restored the etcd metadata cluster
   • recreated the missing Kubernetes objects so the retained PersistentVolumes became available, then recreated the StatefulSets
   • restored the Zookeeper volumes from an hourly Velero backup
   • rebuilt the Kafka StatefulSet one pod at a time with health and readiness checks disabled
   • accepted writes into the Kafka write-ahead log and scaled up the write path with more replicas and higher memory requests and limits
   • recreated the storage pods, which downloaded data from object storage, reindexed it and replayed the write-ahead log
   • re-enabled Argo CD, queries, dashboards and scheduled tasks, restored replica counts and ran final QA checks

   Action items

   • Write runbooks for restoring state when Kubernetes objects such as PersistentVolumeClaims are deleted but the data on the underlying volumes is preserved
   • Set every volume in every environment to retain data when its PersistentVolumeClaim is deleted
   • Improve the process for handling public-facing incidents
   • Store generated YAML as one object per file, so an overwrite of an existing object shows up as such in review
   • Detect duplicate objects when generating YAML and warn before review, treating objects with the same group, namespace and name as the same even when the apiVersion string differs
   • Add the Argo CD Prune=false annotation to all stateful resources and to Namespace objects managed by Argo CD
   • Set FailOnSharedResource=true on Argo CD Applications so Argo CD fails before changing an object another Application manages or managed

   Lessons

   The biggest gap was that monitoring did not notice the deletion quickly, which pushed the team toward a reflex rollback; in a stateful system a revert can create a new, empty instance instead of bringing the old one back. Retained volumes and clients that retry writes are what kept data loss at zero. Rendering many objects into one file let an overwrite of an existing object look like an addition, so generated configuration needs one object per file and duplicate detection that compares objects the way Kubernetes does. Stateful resources need protection from pruning, and that protection must cover their Namespace too, or deleting the namespace removes everything inside it.

   View case

By root cause3