Incident Library

Applied filters Root causeFaulty deploymentOrganisationOpenAIClear allEvery case under the filters, with no sentence; newest first.

Suggested filters Search, and the values found in your words appear here.

Graph

Root cause
Component
Symptom
Contributing factor
Case

Cases1 case

1 to 1 of 1

IL-0121 A faulty code change to a control plane cluster meant for low-priority workloads degraded the production inference workloads it also managed
OpenAI2025
Symptoms
Between 2:10 PM and 3:05 PM PDT on July 9, 2025, success rates of the gpt-4.1-mini model in the API dropped for about 20 minutes, and ChatGPT search returned many more errors than usual for about 35 minutes.
High error rate
Trigger
A faulty code change was deployed to a control plane cluster whose main job is managing low-priority workloads.
Root cause
Faulty deployment
Component
–
Contributing factor
Excessive blast radius No safeguard on destructive change
Remediation
rolled back the faulty code change
Action items
Add safeguards that prevent production inference workloads from being managed by the low-priority control plane cluster
Strengthen testing for deploying high-impact control plane changes
Move all existing production inference workloads under the production control plane cluster
Build a circuit breaker that stops large-scale unintended changes
View case

Check your cluster for this

This case involves conditions a cluster can be checked for. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Faulty deployment1 case

Component: –
Contributing factor: Excessive blast radius No safeguard on destructive change
Organisation: OpenAI

IL-0121 A faulty code change to a control plane cluster meant for low-priority workloads degraded the production inference workloads it also managed
OpenAI2025