Cases4 cases

1 to 4 of 4

1. IL-0235 Azure OpenAI GPT-5.2 returned HTTP 400 in seven regions after a configuration update the model engine did not support

   Microsoft Azure2026

   Symptoms

   From 23:20 UTC on 9 March 2026 Azure OpenAI Service customers received HTTP 400 and HTTP 429 error responses for the GPT-5.2 model in Australia East, Central US, East US 2, Korea Central, Norway East, Sweden Central and UK South. All other GPT models kept working.

   High error rateLoad imbalance

   Trigger

   An update to the Azure OpenAI GPT-5.2 model enabled feature settings meant to improve service efficiency and resilience. The model engine version running in production did not yet support those settings. The update was rolled out to the whole fleet in stages under Safe Deployment Practices.

   Root cause

   Configuration change

   Component

   Model inference serverModel request routing

   Contributing factor

   Unrepresentative stagingMonitoring blind spotOverlapping failures

   Detection

   Service monitoring detected the issue at 00:19 UTC on 10 March, about an hour after impact began. Detection was late because monitoring was set up for client-side HTTP 400 errors and not for service-side anomalies.

   Remediation

   • Rolled back the GPT-5.2 configuration in all affected regions instead of waiting for a hotfix
   • Completed the routing updates and restored full regional capacity information so traffic was spread normally again

   Action items

   • Train engineers on the operating procedures, including which scenarios can be rolled back quickly
   • Store a cached known-good version of the traffic routing details for when the capacity service cannot provide current data
   • Set thresholds for service-side HTTP 400 errors in monitoring
   • Improve anomaly detection on HTTP 4xx errors to alert on unusual error rates below the usual thresholds
   • Add signals to the deployment systems so problematic rollouts stop automatically
   • Make sure early rollout stages have enough backend model instances to catch such problems

   View case
2. IL-0352 Heroku Router 2.0 spread HTTP requests unevenly across dynos for four days, causing latency and timeouts

   Heroku2025

   Symptoms

   From about 22:00 UTC on 18 July 2025, some apps on Heroku's Common Runtime that used Router 2.0 had their HTTP requests balanced unevenly across their web dynos. The dynos that got more requests were overloaded, and the apps saw higher HTTP request latency, timeouts, reduced availability and various Heroku error codes in application logs and dashboards.

   Load imbalanceHigh latencyIntermittent timeouts

   Trigger

   From 22:00 UTC on 18 July 2025 router instances began hitting a code condition that broke the entropy metadata Router 2.0 uses to choose dynos. The condition had been in the code since the Router 2.0 beta and became more likely as more HTTP requests went through the new router, with about 94% of Common Runtime apps migrated.

   Root cause

   Software defect

   Component

   Cloud load balancer

   Contributing factor

   Monitoring blind spotUnrepresentative staging

   Detection

   No alert fired. Heroku Support received the first customer reports of uneven request balancing at 16:46 UTC on 20 July 2025, and an internal investigation was declared on 22 July.

   Remediation

   • Advised affected customers to disable Router 2.0 and fall back to the legacy router
   • Cycled the production router fleets, which improved load balancing for affected apps
   • Deployed the router fix to staging, then rolled it out to all Common Runtime partitions by 00:12 UTC on 23 July

   Action items

   • Introduced gating and notifications so that load balancing behaviour regressions do not occur
   • Communicate the timeline and send reminders to apps still on the legacy router before automated migrations resume

   View case
3. IL-0211 A Mandrill PostgreSQL shard shut down at transaction ID wraparound and email sending was impaired for about 40 hours

   Mailchimp2019

   Symptoms

   From 05:35 UTC on 4 February 2019 Mandrill, Mailchimp's transactional email service, sent only 80% of queued emails, because one of its PostgreSQL shards, shard4, stopped accepting writes. When Mandrill engineers opened an investigation at 11:58 UTC, they saw a sharply elevated load on shard4.

   Writes rejectedThroughput degradationLoad imbalanceNode disk full

   Trigger

   At 05:35 UTC on 4 February the transaction ID counter of shard4, the busiest Mandrill PostgreSQL shard, reached its upper limit and PostgreSQL shut the database down to protect against wraparound.

   Root cause

   Capacity exhaustion

   Component

   DatabasePostgreSQL vacuum

   Contributing factor

   Ignored early signalNo alert on limitSignal buried in noiseRetry stormMonitoring blind spot

   Detection

   The first exception was logged overnight but was buried by unrelated exceptions in the centralized error reporting. Mandrill engineers opened an investigation at 11:58 UTC and, while looking for the source of the rising load on shard4, found the wraparound at 12:56 UTC.

   Remediation

   • Put shard4 into standalone mode and started VACUUM at 14:16 UTC on 4 February
   • Stopped and restarted the vacuum with settings that made it run faster
   • Started a dump and restore of the database without the Search and Url tables while the vacuum ran
   • Replaced storage volumes on the Mandrill app servers whose disks were filling with queued jobs and logs
   • Moved the job-queue tables from shard4 to other live shards
   • Truncated the Search and Url tables on the locked database at 19:12 UTC on 5 February, after which the vacuum finished in about an hour
   • Started sending from the backlog at 21:36 UTC
   • Added alerting for XID wraparound by 22:40 UTC

   Lessons

   Mailchimp named the factors that slowed its response: knowledge of and access to Mandrill systems and monitoring rested with a few people, Mandrill logging needed a locally writable disk, logs of malformed API requests were hard to separate from real application errors, job health was hard to see, and the disk space problem made partial deployments for triage risky. What helped was trained incident commanders with clear roles (incident commanders in 3-hour rotations, scribes, technical liaisons and points of contact between teams), periodic summaries with a list of recent and open decisions, and taking time in a video meeting to reassess, which led to the truncation. Running in the cloud let the team expand storage quickly, and the team was lucky that the largest table holding up the vacuum held data it could delete.

   View case
4. IL-0296 Transloadit's us-east Redis cluster lost queued jobs and heartbeats, raising WORKER_JOB_ERROR errors for 81 customers

   Transloadit2016

   Symptoms

   Reports came in of more WORKER_JOB_ERROR errors, which tell Transloadit there is a communication problem between its Uploaders, the machines that receive files and orchestrate Assemblies, and its Drones, the machines that run encoding tasks. Many Assemblies became much slower because retries with exponential back-off kicked in, and a few could not be replayed.

   High error rateHigh latencyLoad imbalance

   Trigger

   A steep increase in HLS traffic, together with more Drones online processing the same volume faster, which put more load on the central Redis nodes.

   Root cause

   Capacity exhaustion

   Component

   In-memory cacheDatabase replication

   Contributing factor

   Monitoring blind spotUnverified assumptionHarmful manual action

   Detection

   Reports of an increased number of WORKER_JOB_ERROR errors.

   Remediation

   • Re-added lost messages and relaxed heartbeat monitoring as a stopgap
   • Upgraded Redis to the latest stable version
   • Doubled the Redis cluster to six nodes and then added three more, which made it worse
   • Brought the number of read replicas down to one, which cut traffic and failure rates
   • Capped how far Drones scale up in each region

   Action items

   • Add logging and alarms for approaching saturation
   • Find ways to decentralise the traffic between the Redis nodes and the Drones and Uploaders
   • Add sharding per Uploader on top of the sharding per region

   Lessons

   To handle more traffic Transloadit had to use fewer Redis nodes, not more: each added replica multiplied the replication traffic from the single leader that takes every write, so scaling up the number of nodes was the wrong call. Bandwidth had not been among the monitored Redis metrics.

   View case

By root cause3