Cases45 cases

1 to 5 of 45

1. IL-0058 Argo CD deleted namespaces shared by several applications, leaving about 97 applications stuck Missing and OutOfSync

   Argo CD users2026

   Symptoms

   After a rolling restart of the Argo CD application-controller, namespaces that several applications shared went into the Terminating state. About 97 applications were then stuck in a Missing/OutOfSync deadlock.

   Resources deleted

   Trigger

   A manual bulk sync with force: true from the Argo CD UI, run during the window after an application-controller restart, on an application whose manifests declared a namespace whose app.kubernetes.io/instance label pointed to a different application.

   Root cause

   Software defect

   Component

   Argo CD

   Contributing factor

   No safeguard on destructive changeUnverified assumptionHarmful manual action

   Action items

   • Refile the problem as #29739: a manual force sync hard-replaces shared cluster-scoped resources through a tracking-ID conflict, ignoring Prune=false

   Lessons

   The sync option Prune=false only protects the prune path; it does not stop a force sync from deleting and recreating a resource. When two Argo CD applications declare the same namespace or other cluster-scoped resource, force-syncing the application that does not own the tracking label can delete it at any time.

   View case
2. IL-0124 Capacity changes queued while a circuit breaker was engaged ran all at once when it was released, taking multiple OpenAI inference engines out of service and raising API error rates

   OpenAI2026

   Symptoms

   Between about 4:26 PM and 4:56 PM PST on 4 March 2026, some OpenAI API requests to several models failed or took longer than usual, and some Codex traffic was affected too. The overall API success rate dropped temporarily.

   High error rateHigh latency

   Trigger

   A protective circuit breaker that had paused some scheduling operations was released, and the infrastructure actions queued during the pause were executed at the same time.

   Root cause

   Software defect

   Component

   Automated capacity manager

   Contributing factor

   Automation amplificationChange with a delayed effectNo safeguard on destructive changeMonitoring blind spot

   Remediation

   • manually reassigned infrastructure resources to the affected models
   • brought the removed inference engines back online and reassigned them until error rates returned to normal

   Action items

   • Improve safeguards in the automated capacity management systems so they detect and respond to state drift
   • Improve observability of circuit breakers and scheduling systems so their status and effects are easier to identify
   • Add safeguards so queued actions cannot run in large batches after long pauses
   • Improve monitoring and alerting around capacity reconciliation problems

   Lessons

   Pausing one automated system does not stop the systems that feed it. Changes queued during the pause reflect the state at the time they were queued, and releasing the queue all at once applies those stale decisions together. The fixes OpenAI names are detecting state drift, making circuit breaker status visible, and preventing large batches from executing after a long pause.

   View case
3. IL-0150 A latent race condition in DynamoDB's DNS automation left an empty DNS record for dynamodb.us-east-1.amazonaws.com, and the DynamoDB and EC2 launch failures that followed caused EKS, ECS and Fargate container launch failures and cluster scaling delays in us-east-1

   Amazon Web Services2025

   Symptoms

   At 11:48 PM PDT on 19 October 2025, every system that reached DynamoDB in us-east-1 through the public regional endpoint started getting DNS failures and could not connect. This included customer traffic and internal AWS services that depend on DynamoDB. DynamoDB global table replicas in other regions still answered but fell behind in replication to and from us-east-1.

   DNS resolution failuresHigh error rateScale-up failureIntermittent connection failuresHigh latency

   Trigger

   One DNS Enactor, slowed by unusually high retry delays, applied a much older DNS plan to the regional DynamoDB endpoint just as another Enactor that had applied the newest plan ran its clean-up, which deleted that older plan and with it every IP address of the endpoint.

   Root cause

   Software defect

   Component

   Authoritative DNS servers and recordsApplication databaseCloud load balancer

   Contributing factor

   Automation amplificationRetry stormNo runbook

   Detection

   DNS failures to DynamoDB began immediately at 11:48 PM PDT and the engineering teams of affected AWS services were engaged at once; by 12:38 AM they had identified DynamoDB's DNS state as the source. The later NLB health check problem was detected by monitoring at 6:52 AM.

   Remediation

   • applied temporary mitigations by 1:15 AM that let some internal services reach DynamoDB and repaired key internal tooling
   • restored all DynamoDB DNS information by 2:25 AM through manual operator intervention
   • throttled incoming work and selectively restarted DWFM hosts from 4:14 AM so droplet leases could be re-established
   • reduced the load on Network Manager to bring network configuration propagation times back to normal by 10:36 AM
   • disabled automatic health check failover for NLB at 9:36 AM, and re-enabled it at 2:09 PM after EC2 recovered
   • relaxed the EC2 request throttles from 11:23 AM until all EC2 APIs and launches were normal at 1:50 PM

   Action items

   • Disable the DynamoDB DNS Planner and DNS Enactor automation worldwide
   • Fix the race condition and add protections against applying incorrect DNS plans before re-enabling the DNS automation
   • Add a velocity control to NLB that limits how much capacity a single load balancer can remove when health check failures cause availability zone failover
   • Build an additional EC2 test suite that exercises the DWFM recovery workflow at scale
   • Improve throttling in EC2 data propagation systems so incoming work is rate limited by the size of the waiting queue

   Lessons

   A single DNS record that automation can empty is enough to take a widely used regional endpoint off the network, and every service built on it fails in turn. A check made once at the start of a long operation can be stale by the time the operation writes its result. Recovery of the dependency is not recovery of the dependents: systems that lost state during the outage, such as EC2's droplet leases and network propagation backlog, can collapse under their own catch-up work, and container services that rely on new instance launches keep failing to launch and scale until that work is done.

   View case
4. IL-0104 After a CoreDNS pod was OOMKilled, the talk names three causes: Istio's aggressive DNS cache refresh on one reused client port, kube-proxy not cleaning up conntrack entries of terminated pods, and the 2-minute UDP conntrack timeout

   Cisco ThousandEyes2025

   Symptoms

   On 4 September 2025 a CoreDNS pod in a Cisco ThousandEyes Kubernetes cluster was OOMKilled.

   DNS resolution failuresStale endpoint routingOOMKilled

   Trigger

   A CoreDNS pod was OOMKilled.

   Root cause

   Software defect

   Component

   CoreDNSkube-proxyconntrack tableIstio

   Contributing factor

   Unreviewed defaults

   Lessons

   The closing slide sets three behaviours side by side: Istio's extremely aggressive DNS cache refresh with one client port reused for all DNS requests, kube-proxy's lack of a cleanup mechanism for stale conntrack entries of terminated pods (fixed in Kubernetes v1.32), and conntrack's default bidirectional UDP stream timeout of 2 minutes. The speaker describes Istio as making some very complex things simple but some very simple things complex, and presents Cilium, which looks up Services in maps instead of walking the KUBE-SERVICES chain and can replace kube-proxy entirely with its eBPF data path.

   View case
5. IL-0171 Karpenter node auto-repair replaced EKS GPU nodes about 10 minutes after they became Ready, in a loop, because the Bottlerocket NVIDIA AMI lacked libdcgm.so and set AcceleratedHardwareReady to False

   Inflab2025

   Symptoms

   GPU nodes for an AI video upscaling job were created normally and reached Ready, and then about 10 to 11 minutes later Karpenter treated each one as unhealthy and replaced it. Every new node went the same way, so the GPU pods kept restarting and the upscaling work could not make progress.

   Node recreation loopGPU unavailable

   Trigger

   Enabling Karpenter's node auto-repair feature gate (nodeRepair: true) and then running GPU pods on nodes from the Bottlerocket NVIDIA AMI.

   Root cause

   Software defect

   Component

   KarpenterNode operating system imageGPU driver

   Contributing factor

   Automation amplificationUnverified assumption

   Detection

   The team noticed GPU nodes being replaced repeatedly after running GPU pods, then found the DCGMError message in the node events.

   Remediation

   • Disabled Karpenter node auto-repair by setting the nodeRepair feature gate to false

   Lessons

   Automatic node repair acts on every unhealthy node condition, including AcceleratedHardwareReady, so an image defect that only breaks GPU health monitoring is enough to have every GPU node terminated and recreated in a loop. Before turning on auto-repair, check which conditions it watches and whether the node image reports them correctly. Disabling auto-repair stops the loop at once but is only a temporary measure; installing the missing library or changing the AMI address the cause but cost more on an immutable OS such as Bottlerocket.

   View case

By root cause1