Cases32 cases

1 to 5 of 32

1. IL-0179 With DEBUG logging and synchronous log output, a high-traffic service filled the 64 KB stdout pipe, so request threads blocked in write() while holding database connections and HikariCP ran dry, failing requests in 30-second bursts

   MUSINSA2026

   Symptoms

   The on-call engineer for a high-traffic service was alerted that requests were failing in large batches every 30 seconds.

   High error rateIntermittent timeoutsConnection pool exhaustion

   Trigger

   DEBUG level logging was on in production when traffic surged, raising log volume to about 100 times normal while the service logged synchronously.

   Root cause

   Configuration change

   Component

   Application logging libraryContainer runtimeTelemetry agentApplication database

   Contributing factor

   Unverified assumptionExcessive blast radius

   Detection

   The on-call engineer was alerted that requests were failing in large batches every 30 seconds.

   Action items

   • Replace synchronous logging with TraceAwareAsyncAppender, which captures trace_id and span_id on the calling thread and drops logs instead of blocking when the queue is full (neverBlock=true)
   • Keep the amount of log output at a sustainable level, since an asynchronous queue still fills under sustained overload

   Lessons

   Removing AsyncAppender to keep trace IDs also removed the protection that kept request threads away from stdout back-pressure. A company-wide standard applies the same side effect to dozens of services, so check what a standard rests on and what working behaviour it takes away. ThreadLocal context does not cross into an asynchronous worker, but capturing it on the calling thread before handing the event off keeps both the IDs and the asynchronous output. Under extreme load, dropping log lines is preferable to stopping request handling.

   View case
2. IL-0069 Attackers stole a high-privilege service account token from a production Kubernetes cluster and used it to reach a cryptocurrency exchange's financial systems

   Unnamed cryptocurrency exchange2026

   Symptoms

   Inside the cluster the attacker left these traces: a new pod appeared in the production Kubernetes cluster that was built to expose its mounted service account token; requests authenticated as a broad management service account listed secrets and touched workloads in many namespaces; and a backdoor was placed in a running production pod.

   Unauthorised access or a foreign workload

   Trigger

   A spearphishing attack gave the attacker a foothold on a developer's workstation. The attacker used the developer's active, privileged cloud session to deploy a malicious pod to the production cluster.

   Root cause

   Credential exposure

   Component

   RBAC and service accountsKubernetes API server

   Contributing factor

   Excessive blast radius

   Lessons

   Unit 42 concludes that a single over-privileged identity turned one malicious pod into cluster-wide and then cloud-wide access. Its recommendations are least-privilege RBAC with the Restricted Pod Security Standard, short-lived projected service account tokens bound to the pod's lifetime, runtime monitoring of what containers actually do, and Kubernetes audit logging so that identity-driven changes can be seen and reconstructed.

   View case
3. IL-0187 A brief Spring Cloud Config Server downtime failed every client application's liveness probe, because the actuator health check included the config server, and Kubernetes restarted the pods in a kill loop

   Yonghyun Kim2025

   Symptoms

   While the Spring Cloud Config Server was briefly down for an upgrade, every client application began to answer its liveness probe with a failure saying it could not reach the config server.

   Probe failuresContainer crash loopService outage

   Trigger

   A downtime upgrade of the Spring Cloud Config Server made it stop responding for a short time.

   Root cause

   Dependency failure

   Component

   Liveness and readiness probes

   Contributing factor

   Unreviewed defaultsAutomation amplificationExcessive blast radius

   Lessons

   A liveness probe that includes an external dependency turns a short outage of that dependency into a restart loop of every client. Keep external dependencies out of liveness checks and handle temporary problems with readiness probes. Libraries can add health indicators to Spring Boot's actuator health endpoint automatically, so check what the probe path actually reports. The author ran another Kubernetes environment for two years without liveness probes, relying on strict readiness probes and on the application terminating itself on unrecoverable faults.

   View case
4. IL-0122 An invalid value in a configuration shared by several ChatGPT services put pods into crash loops and raised error rates

   OpenAI2025

   Symptoms

   From 7:43 PM to 8:38 PM PDT on July 15, 2025, a small percentage of ChatGPT users saw more errors in their conversations.

   Container crash loopHigh error rate

   Trigger

   An internal engineer applied a configuration change that contained an invalid value.

   Root cause

   Configuration change

   Component

   –

   Contributing factor

   Excessive blast radiusNo staged rollout

   Remediation

   • reverted the configuration to the correct value
   • started a rolling restart of the affected pods to force them to resynchronise

   Action items

   • Roll out future critical configuration changes for the service to a single cluster first, with automated health guards that stop the rollout if availability drops
   • Move to a service-specific configuration value instead of one shared between services
   • Improve deployment safeguards

   View case
5. IL-0121 A faulty code change to a control plane cluster meant for low-priority workloads degraded the production inference workloads it also managed

   OpenAI2025

   Symptoms

   Between 2:10 PM and 3:05 PM PDT on July 9, 2025, success rates of the gpt-4.1-mini model in the API dropped for about 20 minutes, and ChatGPT search returned many more errors than usual for about 35 minutes.

   High error rate

   Trigger

   A faulty code change was deployed to a control plane cluster whose main job is managing low-priority workloads.

   Root cause

   Faulty deployment

   Component

   –

   Contributing factor

   Excessive blast radiusNo safeguard on destructive change

   Remediation

   • rolled back the faulty code change

   Action items

   • Add safeguards that prevent production inference workloads from being managed by the low-priority control plane cluster
   • Strengthen testing for deploying high-impact control plane changes
   • Move all existing production inference workloads under the production control plane cluster
   • Build a circuit breaker that stops large-scale unintended changes

   View case

By root cause8