Cases3 cases

1 to 3 of 3

1. IL-0366 Cloud Storage uploads and writes in us-central1 slowed and failed with HTTP 503 as traffic piled onto a few backends

   Google Cloud2026

   Symptoms

   From 04:00 US/Pacific on 8 October 2026, Google Cloud Storage upload and write operations in the us-central1 region became slow and failed. Clients mostly saw HTTP 503 UNAVAILABLE errors on WriteObject operations and client-side timeouts.

   High latencyHigh error rateLoad imbalance

   Trigger

   A configuration update to the Cloud Storage load balancing layer turned on a traffic sorting optimization that was meant to deal with local capacity constraints. The recurrence on 9 October came from an incomplete rollback of an upload planner feature.

   Root cause

   Configuration change

   Component

   Object storageDatabase

   Contributing factor

   No headroomOverlapping failures

   Detection

   Google engineering teams monitoring system telemetry detected the problem at 04:50 US/Pacific on 8 October, 50 minutes after it began.

   Remediation

   • Drained traffic from the most overloaded storage backend nodes
   • Deployed a configuration change that disabled the traffic sorting optimization
   • Rolled back a recent update to the metadata database service
   • Redirected traffic and drained affected storage components again during the recurrence
   • Fully reverted the load balancing configuration change, restoring capacity-aware routing
   • Returned the drained nodes to service once resource use was back to normal

   Action items

   • Validate load balancing logic changes in smaller regions before rolling them out to large, highly utilized regions
   • Make the storage load balancing enforce capacity constraints so traffic cannot bypass pre-spilling safeguards
   • Speed up internal data movement within highly utilized storage cells to absorb temporary load spikes

   View case
2. IL-0235 Azure OpenAI GPT-5.2 returned HTTP 400 in seven regions after a configuration update the model engine did not support

   Microsoft Azure2026

   Symptoms

   From 23:20 UTC on 9 March 2026 Azure OpenAI Service customers received HTTP 400 and HTTP 429 error responses for the GPT-5.2 model in Australia East, Central US, East US 2, Korea Central, Norway East, Sweden Central and UK South. All other GPT models kept working.

   High error rateLoad imbalance

   Trigger

   An update to the Azure OpenAI GPT-5.2 model enabled feature settings meant to improve service efficiency and resilience. The model engine version running in production did not yet support those settings. The update was rolled out to the whole fleet in stages under Safe Deployment Practices.

   Root cause

   Configuration change

   Component

   Model inference serverModel request routing

   Contributing factor

   Unrepresentative stagingMonitoring blind spotOverlapping failures

   Detection

   Service monitoring detected the issue at 00:19 UTC on 10 March, about an hour after impact began. Detection was late because monitoring was set up for client-side HTTP 400 errors and not for service-side anomalies.

   Remediation

   • Rolled back the GPT-5.2 configuration in all affected regions instead of waiting for a hotfix
   • Completed the routing updates and restored full regional capacity information so traffic was spread normally again

   Action items

   • Train engineers on the operating procedures, including which scenarios can be rolled back quickly
   • Store a cached known-good version of the traffic routing details for when the capacity service cannot provide current data
   • Set thresholds for service-side HTTP 400 errors in monitoring
   • Improve anomaly detection on HTTP 4xx errors to alert on unusual error rates below the usual thresholds
   • Add signals to the deployment systems so problematic rollouts stop automatically
   • Make sure early rollout stages have enough backend model instances to catch such problems

   View case
3. IL-0566 Hot ranges after a Cookie Run: Kingdom guild update drove CockroachDB nodes to 100% CPU, forcing emergency maintenance

   Devsisters2021

   Symptoms

   About half an hour after the Cookie Run: Kingdom guild update maintenance ended at 11:00 on 6 March 2021, one CockroachDB node was using all 24 cores of its 24-core machine and server metrics began to show a significant number of errors. Soon two more nodes showed abnormal CPU use.

   Load imbalanceDatabase overloadHigh error rate

   Trigger

   During the guild update maintenance, new guild tables were added and pre-split across the cluster, and a new database node was added to replace one removed a few days earlier for a physical hardware issue. When the game reopened, all players at once wrote to existing tables whose primary-key prefixes were new or had changed in the update.

   Root cause

   Faulty deployment

   Component

   Database

   Contributing factor

   Unverified assumptionOverlapping failures

   Detection

   Monitoring after the maintenance showed one node's CPU at the ceiling together with errors in server metrics. Later a Slack alert reported a node failing its Host Status Check, and the abnormal-log channel reported filemap_fdatawait_range_keep_errors.

   Remediation

   • Blocked new logins without kicking players already in the game, using the maintenance switch as a makeshift access queue
   • Put the game into full emergency maintenance at 14:50 when several nodes failed health checks
   • Backed up the game data table by table after the full database backup stalled
   • Paused writing play records to reduce load on the database before reopening
   • Split the problem tables from about 18:30
   • Decommissioned the node that failed its Host Status Check
   • Reopened the game and used the login-block queue again when a node reached 100% CPU
   • Terminated the node that kept returning to 100% CPU

   Lessons

   In a single-server relational database a change to the primary-key prefix is usually harmless, but in CockroachDB's distributed design such a change on a production cluster needs great care. The team had studied this in CockroachDB documentation, but living through it taught far more than reading about it.

   View case

By root cause2