Cases5 cases

1 to 5 of 5

1. IL-0448 Requests from Uzabase's Kubernetes Job pods to an LLM agent randomly never got responses when they ran past 15 minutes

   Uzabase2026

   Symptoms

   At Uzabase, a product team in the Speeda business uses an LLM agent to extract companies' management issues from public documents, and a client pod run as a Kubernetes Job sends the requests to the agent. The agent reviews its output many times, so a response often takes more than 15 minutes. Some share of the requests never got a response, with no clear pattern in which ones.

   Intermittent timeouts

   Trigger

   LLM agent requests that took more than 15 minutes without packets on the TCP connection.

   Root cause

   DNS or network fault

   Component

   kube-proxy

   Contributing factor

   Unreviewed defaultsUnknown limit

   Detection

   Client Job pods were not getting responses for some requests to the agent; the client log showed tasks that started but never finished until the 60-minute forced retry.

   Remediation

   • Turned on TCP keepalive in the Python client so packets keep the connection alive

   Lessons

   Application engineers also need to know the network and infrastructure. As AI applications with long response times become common, a silent cut of a long request can come from a network setting such as the IPVS TCP timeout.

   View case
2. IL-0155 Some DOKS clusters in NYC1 saw intermittent DNS timeouts and NodeNotReady events because CoreDNS ran on shared-CPU Droplet worker nodes

   DigitalOcean2026

   Symptoms

   From 20:31 UTC on July 9, 2026, DigitalOcean reported a problem with Kubernetes deployments in its NYC1 region. Application workloads could see intermittent DNS failures and NodeNotReady events.

   DNS resolution failuresIntermittent timeoutsNode NotReady

   Root cause

   Capacity exhaustion

   Component

   CoreDNSNode pool

   Contributing factor

   Unknown limit

   Lessons

   Cluster DNS is sensitive to latency, and CoreDNS on shared-CPU nodes can time out intermittently. DigitalOcean's advice is to place CoreDNS on dedicated-CPU node pools and to run enough CoreDNS replicas.

   View case
3. IL-0048 DNS lookups timed out in a large EKS cluster when CoreDNS hit the 1024 packets-per-second link-local limit, driven by sudo hostname lookups

   unknown2025

   Symptoms

   CI jobs failed now and then because a hostname could not be resolved. The failures came from tensorstore, appeared on several different instances and did not repeat consistently. The error read: "./tensorstore/util/result.h:420: Status not ok: status(): UNAVAILABLE: CURL error Could not resolve hostname: Could not resolve host: company-bucket.s3.amazonaws.com".

   DNS resolution failuresIntermittent timeouts

   Trigger

   An administrative pod running with host networking called sudo in a loop. Each sudo call looked up the node's own hostname, and with ndots:5 and four search domains every lookup became five queries, for both A and AAAA records.

   Root cause

   Capacity exhaustion

   Component

   CoreDNSClient-side DNS resolver and cacheVPC DNS resolver

   Contributing factor

   Unknown limitUnreviewed defaults

   Detection

   Reports of flaky CI jobs that could not resolve an S3 bucket hostname, followed by i/o timeout errors in the CoreDNS logs.

   Remediation

   • added 'Defaults !fqdn' in /etc/sudoers.d/00-no-fqdn (mode 440) to the container image so sudo stopped looking up the hostname

   Lessons

   Traffic from CoreDNS to the VPC DNS resolver is limited to 1024 packets per second per interface, and drops show up only as UDP timeouts; the linklocal_allowance_exceeded counter in ethtool shows them. Names that are not fully qualified are multiplied by the search domains under ndots:5, and AAAA lookups double them again. sudo looks up the host's own name on every call unless 'Defaults !fqdn' is set.

   View case
4. IL-0445 Chatwork's service on EKS became unstable as conntrack tables on some nodes overflowed and connections were dropped

   Chatwork2021

   Symptoms

   In 2020 Chatwork moved its legacy system from EC2 to EKS, running both sides in parallel with Route53 weighted records. Once more than a certain share of requests was sent to the EKS side, the service suddenly became unstable. On some nodes the conntrack table had gone past its limit and connections to the application were being dropped.

   Intermittent connection failuresDNS resolution failuresIntermittent timeoutsLoad imbalance

   Trigger

   During the migration from EC2 to EKS, Chatwork sent more than a certain share of requests to the EKS side. Later, applying node-local-dns with force_tcp in its configuration made DNS unstable again.

   Root cause

   Capacity exhaustion

   Component

   conntrack tableCoreDNSkube-proxyClient-side DNS resolver and cacheVPC DNS resolver

   Contributing factor

   Unreviewed defaultsUnrepresentative stagingUnknown limit

   Remediation

   • Raised the conntrack maxPerCore value in the kube-proxy configuration
   • Installed dns-autoscaler so CoreDNS ran with more pods
   • Installed node-local-dns as a DaemonSet on every node
   • Replaced force_tcp with prefer_udp in the node-local-dns configuration

   Lessons

   On EKS, dns-autoscaler and node-local-dns made DNS stable, but node-local-dns must not use force_tcp there. Stable DNS also matters for sizing the number of pods correctly, since blocked DNS queries had made the pod count grow.

   View case
5. IL-0012 CPU limits and an unpatched kernel CFS quota bug throttled multi-threaded services into stalls, failed health checks and timeouts

   Omio2020

   Symptoms

   For about six months, applications on Omio's Google Kubernetes Engine clusters stalled at random: they got stuck, stopped answering health check requests or lost network connections, and nothing the team looked at explained why.

   CPU throttlingProbe failuresHigh error rateHigh latencyIntermittent timeoutsIntermittent connection failures

   Trigger

   Multi-threaded workloads running with CPU limits on nodes whose kernel did not yet have the fix for a known CFS quota bug that causes unnecessary throttling.

   Root cause

   Software defect

   Component

   Container CPU limitLinux kernelkubelet

   Contributing factor

   Unknown limitMonitoring blind spotUnpatched known bug

   Detection

   Months of random stalls and failed health checks led the team to read the cgroup cpu.stat throttling counters inside the pods, which showed far more throttling than the limits should allow.

   Remediation

   • removed CPU limits until the kernel with the CFS bug fix was deployed across all clusters

   Action items

   • Deploy the kernel with the CFS quota bug fix across all clusters
   • Monitor utilisation, saturation and error metrics, API latency and error rates to confirm the change behaves as expected

   Lessons

   A CPU limit is a time quota per 100 ms period, not a share of cores, so a multi-threaded container can be stopped for most of each period long before it uses the cores it was given, and a kernel bug can make that worse. Throttling is invisible in standard Kubernetes metrics and has to be read from the cgroup counters. Removing limits trades away protection against a container that takes a whole node, which is only acceptable when runtimes are tuned.

   View case

By root cause3