Cases4 cases

1 to 4 of 4

1. IL-0179 With DEBUG logging and synchronous log output, a high-traffic service filled the 64 KB stdout pipe, so request threads blocked in write() while holding database connections and HikariCP ran dry, failing requests in 30-second bursts

   MUSINSA2026

   Symptoms

   The on-call engineer for a high-traffic service was alerted that requests were failing in large batches every 30 seconds.

   High error rateIntermittent timeoutsConnection pool exhaustion

   Trigger

   DEBUG level logging was on in production when traffic surged, raising log volume to about 100 times normal while the service logged synchronously.

   Root cause

   Configuration change

   Component

   Application logging libraryContainer runtimeTelemetry agentApplication database

   Contributing factor

   Unverified assumptionExcessive blast radius

   Detection

   The on-call engineer was alerted that requests were failing in large batches every 30 seconds.

   Action items

   • Replace synchronous logging with TraceAwareAsyncAppender, which captures trace_id and span_id on the calling thread and drops logs instead of blocking when the queue is full (neverBlock=true)
   • Keep the amount of log output at a sustainable level, since an asynchronous queue still fills under sustained overload

   Lessons

   Removing AsyncAppender to keep trace IDs also removed the protection that kept request threads away from stdout back-pressure. A company-wide standard applies the same side effect to dozens of services, so check what a standard rests on and what working behaviour it takes away. ThreadLocal context does not cross into an asynchronous worker, but capturing it on the calling thread before handing the event off keeps both the IDs and the asynchronous output. Under extreme load, dropping log lines is preferable to stopping request handling.

   View case
2. IL-0122 An invalid value in a configuration shared by several ChatGPT services put pods into crash loops and raised error rates

   OpenAI2025

   Symptoms

   From 7:43 PM to 8:38 PM PDT on July 15, 2025, a small percentage of ChatGPT users saw more errors in their conversations.

   Container crash loopHigh error rate

   Trigger

   An internal engineer applied a configuration change that contained an invalid value.

   Root cause

   Configuration change

   Component

   –

   Contributing factor

   Excessive blast radiusNo staged rollout

   Remediation

   • reverted the configuration to the correct value
   • started a rolling restart of the affected pods to force them to resynchronise

   Action items

   • Roll out future critical configuration changes for the service to a single cluster first, with automated health guards that stop the rollout if availability drops
   • Move to a service-specific configuration value instead of one shared between services
   • Improve deployment safeguards

   View case
3. IL-0118 A global change to Kubernetes namespace labels made the networking layer recompute metadata and overwhelmed the control planes of three of OpenAI's largest GPU clusters

   OpenAI2024

   Symptoms

   From 10:15 am PT on 25 November 2024 a large share of both API and ChatGPT traffic failed with timeouts and 503 errors.

   High error rateHigh latencyControl plane unresponsive

   Trigger

   A global change to Kubernetes namespace labels caused the networking layer to recompute metadata. The change had rolled out to the small clusters without problems.

   Root cause

   Configuration change

   Component

   Kubernetes API server

   Contributing factor

   Excessive blast radiusLimited debug access

   Remediation

   • moved workloads out of the impacted clusters by hand
   • with the cloud provider, scaled up the number of control plane nodes while scaling down the number of data plane nodes, which let the control planes recover

   Action items

   • Lock namespace changes so this incident cannot recur
   • Freeze certain types of infrastructure deploys
   • Allowlist identities to prevent a recurrence of this class of incidents
   • Improve the GPU load balancing logic so traffic is re-routed automatically and more effectively when possible
   • Improve change management for fleet-wide infrastructure changes

   View case
4. IL-0035 Worker nodes went NotReady after a new ElastAlert rule without memory limits used up node memory

   NU.nl2019

   Symptoms

   The NU.nl development team got reports that several services on one of its Kubernetes clusters behaved erratically: occasional error pages, slow responses and timeouts. The affected services were mostly internal tools and dashboards.

   Node NotReadyHigh error rateHigh latencyIntermittent timeoutsMonitoring outageOOMKilled

   Trigger

   A new ElastAlert rule, deployed shortly before from the command line, that fired if the Cloudfront-* Elasticsearch indexes stopped receiving documents; those indexes take a few million requests per hour, and the rule made ElastAlert's memory grow without bound.

   Root cause

   Configuration change

   Component

   Container memory limitkubeletPrometheus

   Contributing factor

   No resource requests or limitsUnreviewed defaultsNo alert on limitMonitoring blind spotExcessive blast radius

   Detection

   Users reported error pages, slow responses and timeouts from several services; no alert fired.

   Remediation

   • terminated the NotReady nodes one by one so the auto-scaling group replaced them (the new nodes also went NotReady, but Prometheus and Grafana landed on a stable node)
   • found the ElastAlert pods' memory growth on the Grafana pod memory dashboard
   • fixed the ElastAlert problem that caused the memory growth

   Action items

   • Define alerts on resource usage such as CPU, memory and disk space
   • Keep researching an alerting strategy that combines Elasticsearch, Prometheus Alertmanager and New Relic
   • Consider exporting metrics out of the cluster and running Grafana outside it, on EC2 or a separate Kubernetes cluster
   • Either visualise the important metrics in the ELK stack or make Prometheus and Grafana more available, and improve the metrics strategy
   • Set resource limits for ElastAlert through the Helm values
   • Configure namespaces with default memory requests and limits
   • Isolate customer-facing workloads from build, logging and monitoring workloads (a dedicated node group, or later a separate tools cluster)
   • Deploy everything through automation such as Jenkins pipelines, and consider a GitOps approach
   • Deploy through a pipeline that includes a smoke test step
   • Make Kubernetes and cloud infrastructure work part of the team's sprints, picked up by all team members while pairing with more experienced members, and hold workshops that go deep into specific topics

   Lessons

   Fixing the root cause covers only a small part of what an incident can teach; the chain of smaller failures around it is where most of the learning is. A pod without a memory limit can take whole nodes down, so limits need defaults at the namespace level. Monitoring that runs inside the cluster it watches goes blind with it. Changes installed by hand from the command line leave the rest of the team without a record of what changed.

   View case

By root cause1