Incident Library
Applied filters ComponentAutomated capacity managerFactorMonitoring blind spotClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0124 Capacity changes queued while a circuit breaker was engaged ran all at once when it was released, taking multiple OpenAI inference engines out of service and raising API error rates
    OpenAI2026
    Symptoms

    Between about 4:26 PM and 4:56 PM PST on 4 March 2026, some OpenAI API requests to several models failed or took longer than usual, and some Codex traffic was affected too. The overall API success rate dropped temporarily.

    High error rateHigh latency
    Trigger

    A protective circuit breaker that had paused some scheduling operations was released, and the infrastructure actions queued during the pause were executed at the same time.

    Root cause
    Software defect
    Component
    Automated capacity manager
    Contributing factor
    Automation amplificationChange with a delayed effectNo safeguard on destructive changeMonitoring blind spot
    Remediation
    • manually reassigned infrastructure resources to the affected models
    • brought the removed inference engines back online and reassigned them until error rates returned to normal
    Action items
    • Improve safeguards in the automated capacity management systems so they detect and respond to state drift
    • Improve observability of circuit breakers and scheduling systems so their status and effects are easier to identify
    • Add safeguards so queued actions cannot run in large batches after long pauses
    • Improve monitoring and alerting around capacity reconciliation problems
    Lessons

    Pausing one automated system does not stop the systems that feed it. Changes queued during the pause reflect the state at the time they were queued, and releasing the queue all at once applies those stale decisions together. The fixes OpenAI names are detecting state drift, making circuit breaker status visible, and preventing large batches from executing after a long pause.

    View case
Check your cluster for this

This case involves Automated capacity manager. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Software defect1 case

Component
Automated capacity manager
Contributing factor
Automation amplificationChange with a delayed effectMonitoring blind spotNo safeguard on destructive change
Organisation
OpenAI
  1. IL-0124 Capacity changes queued while a circuit breaker was engaged ran all at once when it was released, taking multiple OpenAI inference engines out of service and raising API error rates
    OpenAI2026