Cases18 cases

6 to 10 of 18

1. IL-0119 ChatGPT web service pods crash-looped on failing health checks after a Cosmos DB failure, leaving too few healthy pods to serve the web layer

   OpenAI2025

   Symptoms

   From 3:30am to 7:08am PT, chatgpt.com and a small number of OpenAI APIs returned elevated error rates. The status page classed the incident as a full outage.

   High error rateContainer crash loopProbe failures

   Trigger

   A failure of Cosmos DB in ChatGPT, at OpenAI's service provider.

   Root cause

   Cloud provider outage

   Component

   Application database

   Contributing factor

   Health check tied to a dependency

   Remediation

   • the service provider recovered the affected Cosmos DB databases at 4:25am PT
   • fixed the web server's health checking logic, which ended the pod crash loop

   Action items

   • Change the web systems to degrade more gracefully when they fail, including concurrency control and better load shedding
   • Make the database layer more resilient to service provider outages (described as already underway)

   Lessons

   The database came back at 4:25am PT, yet users saw errors until 7:08am PT: the web tier's own health checks kept removing pods after the dependency had recovered. The planned changes target the web tier's behaviour under failure, with concurrency control and load shedding, rather than the database alone.

   View case
2. IL-0117 A scheduled maintenance and an ingress upgrade on OpenAI's user-facing clusters caused a networking control plane regression that briefly broke the data plane, and the services that restarted then overwhelmed a backend persistence store with a heavy first-start query

   OpenAI2024

   Symptoms

   On August 16, 2024 from 11:38 AM to 1:16 PM PT, the OpenAI primary API was unreliable: ChatGPT conversations succeeded less often, and logging in and creating accounts were affected. The status page called it elevated error rates for ChatGPT and the Platform API. The incident came in two waves, of 44 and 15 minutes.

   High error rateDatabase overload

   Trigger

   A scheduled maintenance combined with an upgrade to the ingress of OpenAI's user-facing clusters.

   Root cause

   Breaking change in an upgrade

   Component

   Ingress controllerApplication database

   Contributing factor

   No staged rolloutAutomation amplification

   Action items

   • Mitigate the networking control plane regression and validate that control plane restarts do not interfere with the clusters' data plane
   • Change service software to start faster and remove the first-start query, relieving the persistence layer
   • Deploy configuration changes that optimise the networking control plane's effect on the clusters' ability to handle traffic
   • Remove the expensive database query from critical startup paths
   • Add monitoring and alerting for networking control plane issues
   • Introduce staged rollouts with longer soak time for infrastructure changes
   • Audit systems for other slow queries that may affect service start time

   Lessons

   A brief connectivity loss turned into a long outage because recovery itself was expensive: the services that restarted overwhelmed one backend persistence store with a heavy first-start query. Keeping costly queries off startup paths shortens recovery, and staged rollouts with longer soak time limit how many systems an infrastructure regression reaches before it is caught.

   View case
3. IL-0024 Expensive cache-backed API calls and uncancelled database requests overloaded MongoDB and took the Hugging Face Hub down three times

   Hugging Face2024

   Symptoms

   At 08:45 CET on 22 April 2024 the Hugging Face Hub became unstable, and by 09:30 it was unresponsive. For most users the website was slow or could not be reached at all.

   Service outageHigh latencyDatabase overload

   Trigger

   Bursts of requests to revision endpoints that read heavily from the *_info_cache collections: a call to https://huggingface.co/api/datasets/lighteval/mmlu/revision/main rose sharply right before the first crash, and a call to https://huggingface.co/api/models/sentence-transformers/all-mpnet-base-v2/revision/main, a model with 300+ associated spaces, lined up with the crash at 21:30 on 23 April.

   Root cause

   Software defect

   Component

   Application databaseModel hubCluster autoscaler

   Contributing factor

   No headroomRetry stormOverlapping failures

   Detection

   Instability started at 08:45 on 22 April and the Hub was unresponsive by 09:30; the team followed MongoDB Atlas metrics and a MongoDB support case. The root cause was found days later from a new graph of requests per URL.

   Remediation

   • Added a firewall rule blocking all requests with a Java/* user agent
   • Disabled the quicksearch feature on the Hub
   • Scaled the MongoDB Atlas cluster by hand from tier 2 to tier 3
   • Switched the Hub to maintenance mode with a 503 page through AWS WAF and blocked incoming traffic
   • Scaled all Hub pods down to 0 and brought them back
   • Set an aggressive rate limit
   • Added timeouts to requests and reworked database queries that were slow or returned large bodies

   Action items

   • Add timeouts to requests based on their usual completion time
   • Optimise requests that took too long or returned a large body, and rework poorly designed database queries
   • Remove old unused indexes from the database
   • Build new dashboards and monitoring metrics
   • Move the *_info_cache collections to Redis
   • Make it possible to abort any database request when the client aborts theirs

   Lessons

   Finding the root cause took two to three days: the existing charts did not show it, and a per-URL request breakdown did. Not cancelling database work when a client goes away turns a strained database into a thundering herd. A close inspection of the database exposed architectural flaws and bottlenecks, such as a collection with large unused indexes, and taught the team which metrics need closer watching to anticipate crashes. Unrelated failures happening at the same time added to the stress and made restoring service harder.

   View case
4. IL-0110 A Grafana Agent pod's high memory use and crash-looping Hangfire pods drove a Kubernetes node into system OOM and NotReady, twice in one day

   DFDS2024

   Symptoms

   At 10:01 UTC on 2024-03-13 a Kubernetes node entered NotReady state, with CPU requests at 78% and memory requests at 82%. An alert fired in Slack at 10:03 and was picked up at 10:05. An attempt to SSH into the node at 10:13 failed because it was unresponsive; it started responding again at 10:19.

   Node NotReadyContainer crash loopOOMKilledMonitoring outage

   Trigger

   Pods of a dev container that used Hangfire to trigger jobs failed with OOM and crash-looped on a node where a Grafana Agent pod was already using a lot of memory.

   Root cause

   Capacity exhaustion

   Component

   Telemetry agentContainer memory limitApplication database

   Contributing factor

   Excessive blast radiusNo resource requests or limitsHarmful manual actionMonitoring blind spot

   Detection

   A Slack alert fired two minutes after the node went NotReady and was picked up two minutes later; the second node's NotReady at 15:03 also raised a Slack alert.

   Remediation

   • marked the EC2 instance unhealthy so a new node started
   • cordoned and force-drained the NotReady node and terminated its EC2 instance
   • removed the new LimitRange from the Grafana Agent namespace and restarted the Grafana Agent pod
   • scaled the dev container to 0 replicas, after which EC2 network traffic returned to normal
   • after the second NotReady, scaled the dev container to 0 replicas again and added a new node
   • emailed the team that owns the dev container, which confirmed it was working on a fix

   Action items

   • Move grafana-agent to the monitoring node group
   • Add alerts for when bandwidth across the cluster reaches a certain threshold
   • Investigate whether memory limits can be set on the grafana-agent Helm chart
   • Investigate grafana-agent memory usage
   • Ask Grafana support whether grafana-agent should use a PVC
   • Consider pushing node logs to Loki with limited retention
   • Create a wiki page with common metrics and log queries to check when investigating issues with Kubernetes nodes

   Lessons

   Dev workloads mixed with production let one crash-looping dev container take down nodes that production also used, and a monitoring agent outside its dedicated node group added to the memory pressure. Scaling the suspect container back up before its owners had fixed it brought the failure back on the replacement node. The team also noted that action items from earlier post-mortems need follow-up, and that knowledge of useful CloudWatch metrics had not been shared, which made the diagnosis harder.

   View case
5. IL-0202 A burst of queries and database writes exhausted connection pools and memory on Honeycomb's ingest hosts, and Kubernetes crash-loop back-off kept failed hosts offline so ingest capacity stayed too low to recover

   Honeycomb2023sev1

   Symptoms

   Standard alerting paged Honeycomb's on-call engineers about the performance and stability of event ingestion, and the automated load-shedding mechanisms tripped and sent further notifications. The status page described a higher than normal error rate and response time on the ingestion service.

   High error rateHigh latencyContainer crash loopService outageDatabase overload

   Trigger

   An abnormally large number of queries arrived from many separate connections within a few minutes.

   Root cause

   Capacity exhaustion

   Component

   Application databaseIn-memory cache

   Contributing factor

   Retry stormAutomation amplification

   Detection

   Standard alerting notified the on-call engineers about ingestion performance and stability, and the automated load-shedding mechanisms added notifications when they tripped. The first status post, about alerts on a higher than normal error rate and response time, went up at 18:28 PST (02:28 UTC), 13 minutes after the 02:15 UTC start.

   Remediation

   • Scaled the ingest fleet up manually and aggressively
   • Cut off most ingest for a few minutes so the ingest hosts could come back with enough capacity
   • Temporarily lowered the rate limits for the most aggressive traffic sources until their traffic subsided

   Action items

   • Keep the ingest fleet scaled up to prevent a repeat while longer-term fixes are worked on
   • Decide on the projects that will address these failure paths

   Lessons

   Load shedding and auto-scaling were in place, yet a burst that arrived faster than the cache could spread its writes still took the ingest hosts down. During recovery, Kubernetes' crash loop back-off worked against the operators: the hosts that had failed stayed offline while the fleet needed every host. Clients that retry whole OTel batches after a partial drop can multiply traffic many times over, so per-source rate limits were what finally stabilised ingest.

   View case

By root cause8