Incident Library
Applied filters SymptomService outageFactorNo headroomOrganisationOpenAIClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0123 A response buffer allocated anew in a loop made OpenAI's routing layer nodes hit their memory limits and fail readiness checks on a morning with more completions than any day before, causing 502 and 503 errors across all models
    OpenAI2023
    Symptoms

    Between 5:42 AM and 7:16 AM PT on 8 November 2023 a large share of requests to OpenAI failed with 502 or 503 errors. Every model and API endpoint saw significant failures.

    High error rateService outageProbe failures
    Trigger

    A morning with dramatically more completions than any previous day pushed the routing layer nodes, which already had a chronic memory problem, past their memory limits.

    Root cause
    Software defect
    Component
    Model request routingContainer memory limit
    Contributing factor
    No alert on limitNo headroom
    Remediation
    • limited incoming traffic
    • redeployed the routing service en masse
    • ramped traffic back up slowly
    Action items
    • Pre-allocate the responseBuffer and reuse it instead of allocating a new one in a loop (3X improvement in memory and CPU usage)
    • Adjust the configured memory limits to an appropriate level so the service has significant headroom
    • Add rate limit controls to shed traffic more gracefully
    • Increase the service's capacity
    • Change alerting to catch the underlying memory behaviour before it escalates into service issues
    • Configure autoscaling for the service, which was not possible before because added capacity could hurt an upstream service
    View case
  2. IL-0115 One of OpenAI's three production AKS clusters went offline through a human error, and rebuilding it took 10½ hours because an AKS Kubernetes upgrade broke the ingress and AKS cannot be downgraded
    OpenAI2022
    Symptoms

    At 18:56 PDT on 23 June 2022 (01:56 UTC on 24 June) several models went offline at once: some fine-tuned curie and davinci models, the Codex beta models code-davinci-001 and code-cushman-001, the legacy GPT models curie, babbage and ada, and all Embeddings models.

    Service outageHigh error rateIntermittent timeouts
    Trigger

    A destructive operation on one of the three production clusters, caused by human error that the existing safeguards did not catch, took the whole cluster offline.

    Root cause
    Operator error
    Component
    Ingress controllerModel hub
    Contributing factor
    No safeguard on destructive changeNo headroomDeferred upgradesOverlapping failures
    Remediation
    • re-allocated the models from the lost cluster to the remaining two clusters
    • rebuilt the VPN link with Azure support after losing the connection to AKS during cluster setup
    • built workarounds with Azure support after losing the connection to Azure Private Link
    • tore down the cluster and rebuilt it from scratch after the upgraded Kubernetes version broke the ingress
    • moved traffic back into the rebuilt cluster between 05:30 and 06:30 PDT
    Action items
    • Add checks and locks so that this and similar production-critical infrastructure cannot be modified by accident
    • Add better visibility into the operations being performed and make parts of the system more lexicographically distinct
    • Keep working with Azure on the AKS network connectivity issues so they do not disrupt new cluster operations
    • Update playbooks with the new cluster failure modes and change the setup steps for new clusters and infrastructure
    • Upgrade Kubernetes systematically and set aside time so the clusters do not fall behind again
    • Shard fine-tuned models across multiple clusters and work towards more automatic cross-cluster replication
    • Distribute and replicate Azure Blob Storage so fine-tuned models load faster and more reliably
    • Finish rolling out node-to-node caching to relieve Azure Blob Storage bottlenecks
    • Change how model weights are stored so they are more efficient and faster to load
    Lessons

    Losing one of three clusters exhausted the GPU headroom, and GPUs could not be added on short notice, so full recovery depended on bringing the same cluster back. Rebuilding it exposed a deferred Kubernetes upgrade: the newer version broke the ingress, and because AKS cannot downgrade there was no way back except a full rebuild. Recovering many large models at once is limited by the bandwidth of the storage they load from, and the caches that normally hide this do not help when everything moves to a different cluster.

    View case
Check your cluster for this

These 2 cases involve Ingress controller and Container memory limit. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause2

Software defect1 case

Component
Container memory limitModel request routing
Contributing factor
No alert on limitNo headroom
Organisation
OpenAI
  1. IL-0123 A response buffer allocated anew in a loop made OpenAI's routing layer nodes hit their memory limits and fail readiness checks on a morning with more completions than any day before, causing 502 and 503 errors across all models
    OpenAI2023

Operator error1 case

Component
Ingress controllerModel hub
Contributing factor
No safeguard on destructive changeNo headroomOverlapping failuresDeferred upgrades
Organisation
OpenAI
  1. IL-0115 One of OpenAI's three production AKS clusters went offline through a human error, and rebuilding it took 10½ hours because an AKS Kubernetes upgrade broke the ingress and AKS cannot be downgraded
    OpenAI2022