Incident Library
Applied filters ComponentModel request routingSymptomHigh error rateFactorNo headroomClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0123 A response buffer allocated anew in a loop made OpenAI's routing layer nodes hit their memory limits and fail readiness checks on a morning with more completions than any day before, causing 502 and 503 errors across all models
    OpenAI2023
    Symptoms

    Between 5:42 AM and 7:16 AM PT on 8 November 2023 a large share of requests to OpenAI failed with 502 or 503 errors. Every model and API endpoint saw significant failures.

    High error rateService outageProbe failures
    Trigger

    A morning with dramatically more completions than any previous day pushed the routing layer nodes, which already had a chronic memory problem, past their memory limits.

    Root cause
    Software defect
    Component
    Model request routingContainer memory limit
    Contributing factor
    No alert on limitNo headroom
    Remediation
    • limited incoming traffic
    • redeployed the routing service en masse
    • ramped traffic back up slowly
    Action items
    • Pre-allocate the responseBuffer and reuse it instead of allocating a new one in a loop (3X improvement in memory and CPU usage)
    • Adjust the configured memory limits to an appropriate level so the service has significant headroom
    • Add rate limit controls to shed traffic more gracefully
    • Increase the service's capacity
    • Change alerting to catch the underlying memory behaviour before it escalates into service issues
    • Configure autoscaling for the service, which was not possible before because added capacity could hurt an upstream service
    View case
Check your cluster for this

This case involves Container memory limit and Model request routing. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Software defect1 case

Component
Container memory limitModel request routing
Contributing factor
No alert on limitNo headroom
Organisation
OpenAI
  1. IL-0123 A response buffer allocated anew in a loop made OpenAI's routing layer nodes hit their memory limits and fail readiness checks on a morning with more completions than any day before, causing 502 and 503 errors across all models
    OpenAI2023