Cases3 cases

1 to 3 of 3

1. IL-0121 A faulty code change to a control plane cluster meant for low-priority workloads degraded the production inference workloads it also managed

   OpenAI2025

   Symptoms

   Between 2:10 PM and 3:05 PM PDT on July 9, 2025, success rates of the gpt-4.1-mini model in the API dropped for about 20 minutes, and ChatGPT search returned many more errors than usual for about 35 minutes.

   High error rate

   Trigger

   A faulty code change was deployed to a control plane cluster whose main job is managing low-priority workloads.

   Root cause

   Faulty deployment

   Component

   –

   Contributing factor

   Excessive blast radiusNo safeguard on destructive change

   Remediation

   • rolled back the faulty code change

   Action items

   • Add safeguards that prevent production inference workloads from being managed by the low-priority control plane cluster
   • Strengthen testing for deploying high-impact control plane changes
   • Move all existing production inference workloads under the production control plane cluster
   • Build a circuit breaker that stops large-scale unintended changes

   View case
2. IL-0086 A deployment meant for a CircleCI test Kubernetes cluster also removed Traefik and Calico resources from the production cluster, taking down the CircleCI UI and runner.circleci.com

   CircleCI2021

   Symptoms

   At 15:35 EST on 11 March 2021 the CircleCI web UI and runner.circleci.com became unavailable. The API, which uses a separate ingress point, stayed reachable, and builds kept running.

   Service outageResources deleted

   Trigger

   An update deployed to a test Kubernetes cluster also removed Traefik and Calico resources from CircleCI's primary production cluster.

   Root cause

   Faulty deployment

   Component

   Ingress controllerCalico

   Contributing factor

   Excessive blast radiusNo runbookMonitoring blind spot

   Detection

   CircleCI engineers were alerted and began investigating. The status page posted its first notice at 20:45 UTC.

   Remediation

   • restored Traefik in the production cluster
   • restored Calico in the production cluster, after which pods served customer requests again

   Action items

   • Improve network-level documentation and runbooks to cover every layer of the Kubernetes networks
   • Improve monitoring and alerting for the network infrastructure
   • Add guardrails that isolate the production and staging clusters so they cannot interfere with each other

   Lessons

   A deployment path that can reach both a test cluster and production can remove shared networking components from production. When the ingress controller and the CNI are both affected, restoring the ingress alone is not enough: engineers had to find the link between Calico and Traefik and restore Calico before pods served traffic again. Time went to DNS and AWS red herrings because nothing linked the outage to the test deployment; CircleCI's answer is better network runbooks, network alerting and guardrails between clusters.

   View case
3. IL-0034 A crashing stats agent sidecar marked every pod unready and the service mesh pulled all services out of traffic

   Airbnb2019

   Symptoms

   An operation to upgrade the stats agent in Kubernetes was announced in advance, with a warning that metrics might be unavailable to services for a few minutes. A few minutes later a reply came in saying that all of the sender's Kubernetes services seemed to be down.

   Probe failuresService outage

   Trigger

   A scheduled upgrade of the stats agent sidecar, during which the agent container crashed.

   Root cause

   Faulty deployment

   Component

   Telemetry agentService discovery

   Contributing factor

   Unverified assumptionExcessive blast radius

   Detection

   A reply to the operation announcement, a few minutes later, said that all of the sender's Kubernetes services appeared to be down.

   Action items

   • Mark logging, tracing and stats agent containers as non-critical and leave them out of the service mesh's pod readiness check

   Lessons

   Every container in a pod affects the pod's readiness, so each sidecar needs a deliberate decision about how its health should count. A sidecar that crashes is not ready even when it has no probes, and any logic that requires all containers to be ready will take the pod out of traffic.

   View case

By root cause1