Incident Library
Applied filters Root causeSoftware defectSymptomHigh latencyFactorMonitoring blind spotClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0124 Capacity changes queued while a circuit breaker was engaged ran all at once when it was released, taking multiple OpenAI inference engines out of service and raising API error rates
    OpenAI2026
    Symptoms

    Between about 4:26 PM and 4:56 PM PST on 4 March 2026, some OpenAI API requests to several models failed or took longer than usual, and some Codex traffic was affected too. The overall API success rate dropped temporarily.

    High error rateHigh latency
    Trigger

    A protective circuit breaker that had paused some scheduling operations was released, and the infrastructure actions queued during the pause were executed at the same time.

    Root cause
    Software defect
    Component
    Automated capacity manager
    Contributing factor
    Automation amplificationChange with a delayed effectNo safeguard on destructive changeMonitoring blind spot
    Remediation
    • manually reassigned infrastructure resources to the affected models
    • brought the removed inference engines back online and reassigned them until error rates returned to normal
    Action items
    • Improve safeguards in the automated capacity management systems so they detect and respond to state drift
    • Improve observability of circuit breakers and scheduling systems so their status and effects are easier to identify
    • Add safeguards so queued actions cannot run in large batches after long pauses
    • Improve monitoring and alerting around capacity reconciliation problems
    Lessons

    Pausing one automated system does not stop the systems that feed it. Changes queued during the pause reflect the state at the time they were queued, and releasing the queue all at once applies those stale decisions together. The fixes OpenAI names are detecting state drift, making circuit breaker status visible, and preventing large batches from executing after a long pause.

    View case
  2. IL-0012 CPU limits and an unpatched kernel CFS quota bug throttled multi-threaded services into stalls, failed health checks and timeouts
    Omio2020
    Symptoms

    For about six months, applications on Omio's Google Kubernetes Engine clusters stalled at random: they got stuck, stopped answering health check requests or lost network connections, and nothing the team looked at explained why.

    CPU throttlingProbe failuresHigh error rateHigh latencyIntermittent timeoutsIntermittent connection failures
    Trigger

    Multi-threaded workloads running with CPU limits on nodes whose kernel did not yet have the fix for a known CFS quota bug that causes unnecessary throttling.

    Root cause
    Software defect
    Component
    Container CPU limitLinux kernelkubelet
    Contributing factor
    Unknown limitMonitoring blind spotUnpatched known bug
    Detection

    Months of random stalls and failed health checks led the team to read the cgroup cpu.stat throttling counters inside the pods, which showed far more throttling than the limits should allow.

    Remediation
    • removed CPU limits until the kernel with the CFS bug fix was deployed across all clusters
    Action items
    • Deploy the kernel with the CFS quota bug fix across all clusters
    • Monitor utilisation, saturation and error metrics, API latency and error rates to confirm the change behaves as expected
    Lessons

    A CPU limit is a time quota per 100 ms period, not a share of cores, so a multi-threaded container can be stopped for most of each period long before it uses the cores it was given, and a kernel bug can make that worse. Throttling is invisible in standard Kubernetes metrics and has to be read from the cgroup counters. Removing limits trades away protection against a container that takes a whole node, which is only acceptable when runtimes are tuned.

    View case
Check your cluster for this

These 2 cases involve Automated capacity manager and Container CPU limit. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Software defect2 cases

Component
Automated capacity manager1Container CPU limit1kubelet1Linux kernel1
Contributing factor
Monitoring blind spot2Automation amplification1Change with a delayed effect1Unpatched known bug1Unknown limit1No safeguard on destructive change1
Organisation
Omio1OpenAI1
  1. IL-0124 Capacity changes queued while a circuit breaker was engaged ran all at once when it was released, taking multiple OpenAI inference engines out of service and raising API error rates
    OpenAI2026
  2. IL-0012 CPU limits and an unpatched kernel CFS quota bug throttled multi-threaded services into stalls, failed health checks and timeouts
    Omio2020