Incident Library
Applied filters ComponentApplication databaseSymptomDatabase overloadFactorAutomation amplificationClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0117 A scheduled maintenance and an ingress upgrade on OpenAI's user-facing clusters caused a networking control plane regression that briefly broke the data plane, and the services that restarted then overwhelmed a backend persistence store with a heavy first-start query
    OpenAI2024
    Symptoms

    On August 16, 2024 from 11:38 AM to 1:16 PM PT, the OpenAI primary API was unreliable: ChatGPT conversations succeeded less often, and logging in and creating accounts were affected. The status page called it elevated error rates for ChatGPT and the Platform API. The incident came in two waves, of 44 and 15 minutes.

    High error rateDatabase overload
    Trigger

    A scheduled maintenance combined with an upgrade to the ingress of OpenAI's user-facing clusters.

    Root cause
    Breaking change in an upgrade
    Component
    Ingress controllerApplication database
    Contributing factor
    No staged rolloutAutomation amplification
    Action items
    • Mitigate the networking control plane regression and validate that control plane restarts do not interfere with the clusters' data plane
    • Change service software to start faster and remove the first-start query, relieving the persistence layer
    • Deploy configuration changes that optimise the networking control plane's effect on the clusters' ability to handle traffic
    • Remove the expensive database query from critical startup paths
    • Add monitoring and alerting for networking control plane issues
    • Introduce staged rollouts with longer soak time for infrastructure changes
    • Audit systems for other slow queries that may affect service start time
    Lessons

    A brief connectivity loss turned into a long outage because recovery itself was expensive: the services that restarted overwhelmed one backend persistence store with a heavy first-start query. Keeping costly queries off startup paths shortens recovery, and staged rollouts with longer soak time limit how many systems an infrastructure regression reaches before it is caught.

    View case
  2. IL-0202 A burst of queries and database writes exhausted connection pools and memory on Honeycomb's ingest hosts, and Kubernetes crash-loop back-off kept failed hosts offline so ingest capacity stayed too low to recover
    Honeycomb2023sev1
    Symptoms

    Standard alerting paged Honeycomb's on-call engineers about the performance and stability of event ingestion, and the automated load-shedding mechanisms tripped and sent further notifications. The status page described a higher than normal error rate and response time on the ingestion service.

    High error rateHigh latencyContainer crash loopService outageDatabase overload
    Trigger

    An abnormally large number of queries arrived from many separate connections within a few minutes.

    Root cause
    Capacity exhaustion
    Component
    Application databaseIn-memory cache
    Contributing factor
    Retry stormAutomation amplification
    Detection

    Standard alerting notified the on-call engineers about ingestion performance and stability, and the automated load-shedding mechanisms added notifications when they tripped. The first status post, about alerts on a higher than normal error rate and response time, went up at 18:28 PST (02:28 UTC), 13 minutes after the 02:15 UTC start.

    Remediation
    • Scaled the ingest fleet up manually and aggressively
    • Cut off most ingest for a few minutes so the ingest hosts could come back with enough capacity
    • Temporarily lowered the rate limits for the most aggressive traffic sources until their traffic subsided
    Action items
    • Keep the ingest fleet scaled up to prevent a repeat while longer-term fixes are worked on
    • Decide on the projects that will address these failure paths
    Lessons

    Load shedding and auto-scaling were in place, yet a burst that arrived faster than the cache could spread its writes still took the ingest hosts down. During recovery, Kubernetes' crash loop back-off worked against the operators: the hosts that had failed stayed offline while the fleet needed every host. Clients that retry whole OTel batches after a partial drop can multiply traffic many times over, so per-source rate limits were what finally stabilised ingest.

    View case
Check your cluster for this

These 2 cases involve Application database and In-memory cache. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause2

Breaking change in an upgrade1 case

Component
Application databaseIngress controller
Contributing factor
Automation amplificationNo staged rollout
Organisation
OpenAI
  1. IL-0117 A scheduled maintenance and an ingress upgrade on OpenAI's user-facing clusters caused a networking control plane regression that briefly broke the data plane, and the services that restarted then overwhelmed a backend persistence store with a heavy first-start query
    OpenAI2024

Capacity exhaustion1 case

Component
In-memory cacheApplication database
Contributing factor
Automation amplificationRetry storm
Organisation
Honeycomb
  1. IL-0202 A burst of queries and database writes exhausted connection pools and memory on Honeycomb's ingest hosts, and Kubernetes crash-loop back-off kept failed hosts offline so ingest capacity stayed too low to recover
    Honeycomb2023