Cases2 cases

1 to 2 of 2

1. IL-0179 With DEBUG logging and synchronous log output, a high-traffic service filled the 64 KB stdout pipe, so request threads blocked in write() while holding database connections and HikariCP ran dry, failing requests in 30-second bursts

   MUSINSA2026

   Symptoms

   The on-call engineer for a high-traffic service was alerted that requests were failing in large batches every 30 seconds.

   High error rateIntermittent timeoutsConnection pool exhaustion

   Trigger

   DEBUG level logging was on in production when traffic surged, raising log volume to about 100 times normal while the service logged synchronously.

   Root cause

   Configuration change

   Component

   Application logging libraryContainer runtimeTelemetry agentDatabase

   Contributing factor

   Unverified assumptionExcessive blast radius

   Detection

   The on-call engineer was alerted that requests were failing in large batches every 30 seconds.

   Action items

   • Replace synchronous logging with TraceAwareAsyncAppender, which captures trace_id and span_id on the calling thread and drops logs instead of blocking when the queue is full (neverBlock=true)
   • Keep the amount of log output at a sustainable level, since an asynchronous queue still fills under sustained overload

   Lessons

   Removing AsyncAppender to keep trace IDs also removed the protection that kept request threads away from stdout back-pressure. A company-wide standard applies the same side effect to dozens of services, so check what a standard rests on and what working behaviour it takes away. ThreadLocal context does not cross into an asynchronous worker, but capturing it on the calling thread before handing the event off keeps both the IDs and the asynchronous output. Under extreme load, dropping log lines is preferable to stopping request handling.

   View case
2. IL-0247 One in a million search requests took 100 times longer on Pinterest's Manas leaf nodes after moving to Kubernetes

   Pinterest2025

   Symptoms

   During performance testing of Pinterest's search system Manas on its new Kubernetes platform, a small but persistent share of search requests timed out. About one request in a million took 100 times longer than usual.

   High latencyIntermittent timeouts

   Root cause

   Software defect

   Component

   Telemetry agentLinux kernel

   Contributing factor

   Unreviewed defaults

   Detection

   Performance testing with traffic sent to a test cluster showed a small but persistent percentage of requests timing out.

   Remediation

   • Disabled cAdvisor's working set size estimation on all PinCompute nodes

   Lessons

   CPU shielding does not fully isolate a process from its neighbours. Showing early that the node image was not the cause narrowed the search and avoided dead ends. Simple blackbox debugging, breaking things until the system stops working, works as a way of bisecting the problem.

   View case

By root cause2