Cases5 cases

1 to 5 of 5

1. IL-0366 Cloud Storage uploads and writes in us-central1 slowed and failed with HTTP 503 as traffic piled onto a few backends

   Google Cloud2026

   Symptoms

   From 04:00 US/Pacific on 8 October 2026, Google Cloud Storage upload and write operations in the us-central1 region became slow and failed. Clients mostly saw HTTP 503 UNAVAILABLE errors on WriteObject operations and client-side timeouts.

   High latencyHigh error rateLoad imbalance

   Trigger

   A configuration update to the Cloud Storage load balancing layer turned on a traffic sorting optimization that was meant to deal with local capacity constraints. The recurrence on 9 October came from an incomplete rollback of an upload planner feature.

   Root cause

   Configuration change

   Component

   Object storageDatabase

   Contributing factor

   No headroomOverlapping failures

   Detection

   Google engineering teams monitoring system telemetry detected the problem at 04:50 US/Pacific on 8 October, 50 minutes after it began.

   Remediation

   • Drained traffic from the most overloaded storage backend nodes
   • Deployed a configuration change that disabled the traffic sorting optimization
   • Rolled back a recent update to the metadata database service
   • Redirected traffic and drained affected storage components again during the recurrence
   • Fully reverted the load balancing configuration change, restoring capacity-aware routing
   • Returned the drained nodes to service once resource use was back to normal

   Action items

   • Validate load balancing logic changes in smaller regions before rolling them out to large, highly utilized regions
   • Make the storage load balancing enforce capacity constraints so traffic cannot bypass pre-spilling safeguards
   • Speed up internal data movement within highly utilized storage cells to absorb temporary load spikes

   View case
2. IL-0287 One overloaded shard of MongoDB's internal monitoring database delayed Atlas metrics, alerts and cluster operations

   MongoDB2025

   Symptoms

   On 7 October 2025 MongoDB reported a delay in the metric ingestion pipeline of MongoDB Atlas and Cloud Manager. Metrics in Atlas dashboards arrived late or went missing, Host Down alerts did not fire, and Atlas cluster management operations such as provisioning and resizing were delayed, while customer clusters themselves stayed healthy and available.

   Monitoring outageDatabase overloadLoad imbalance

   Trigger

   The rollout of an updated internal software version caused a temporary spike in resource demand on the internal database cluster.

   Root cause

   Capacity exhaustion

   Component

   Database

   Contributing factor

   Inefficient queryNo headroom

   Remediation

   • Took corrective actions to stabilise the system and monitored their effect
   • Implemented a fix and monitored the results until the incident was resolved

   Action items

   • Shard the collections with concentrated load to spread traffic over more nodes
   • Optimise inefficient queries
   • Provision additional infrastructure capacity for higher traffic
   • Refine capacity planning to anticipate future load spikes, using metrics from this event
   • Redesign software rollouts to account for the predictable rise in resource demand during a rollout
   • Keep monitoring how the sharding and query changes resolve the hotspots
   • Expand the detection of elevated load conditions for faster anomaly detection and response

   View case
3. IL-0545 Wasabi API calls in all regions returned HTTP 5XX when the message queue stopped spreading requests over database nodes

   Wasabi2024

   Symptoms

   From 00:17 UTC on 30 November 2024, client connections to Wasabi Cloud Storage and the Web Console failed in every storage region, and all API calls returned HTTP 5XX errors.

   Service outageLoad imbalance

   Root cause

   unknown

   Component

   Message brokerDatabase

   Contributing factor

   –

   Remediation

   • Configured internal servers by hand to route requests across multiple database instances, which restored service
   • After service was restored, worked on the root cause by recovering the internal messaging queue and resuming its automated handling of client requests

   View case
4. IL-0566 Hot ranges after a Cookie Run: Kingdom guild update drove CockroachDB nodes to 100% CPU, forcing emergency maintenance

   Devsisters2021

   Symptoms

   About half an hour after the Cookie Run: Kingdom guild update maintenance ended at 11:00 on 6 March 2021, one CockroachDB node was using all 24 cores of its 24-core machine and server metrics began to show a significant number of errors. Soon two more nodes showed abnormal CPU use.

   Load imbalanceDatabase overloadHigh error rate

   Trigger

   During the guild update maintenance, new guild tables were added and pre-split across the cluster, and a new database node was added to replace one removed a few days earlier for a physical hardware issue. When the game reopened, all players at once wrote to existing tables whose primary-key prefixes were new or had changed in the update.

   Root cause

   Faulty deployment

   Component

   Database

   Contributing factor

   Unverified assumptionOverlapping failures

   Detection

   Monitoring after the maintenance showed one node's CPU at the ceiling together with errors in server metrics. Later a Slack alert reported a node failing its Host Status Check, and the abnormal-log channel reported filemap_fdatawait_range_keep_errors.

   Remediation

   • Blocked new logins without kicking players already in the game, using the maintenance switch as a makeshift access queue
   • Put the game into full emergency maintenance at 14:50 when several nodes failed health checks
   • Backed up the game data table by table after the full database backup stalled
   • Paused writing play records to reduce load on the database before reopening
   • Split the problem tables from about 18:30
   • Decommissioned the node that failed its Host Status Check
   • Reopened the game and used the login-block queue again when a node reached 100% CPU
   • Terminated the node that kept returning to 100% CPU

   Lessons

   In a single-server relational database a change to the primary-key prefix is usually harmless, but in CockroachDB's distributed design such a change on a production cluster needs great care. The team had studied this in CockroachDB documentation, but living through it taught far more than reading about it.

   View case
5. IL-0211 A Mandrill PostgreSQL shard shut down at transaction ID wraparound and email sending was impaired for about 40 hours

   Mailchimp2019

   Symptoms

   From 05:35 UTC on 4 February 2019 Mandrill, Mailchimp's transactional email service, sent only 80% of queued emails, because one of its PostgreSQL shards, shard4, stopped accepting writes. When Mandrill engineers opened an investigation at 11:58 UTC, they saw a sharply elevated load on shard4.

   Writes rejectedThroughput degradationLoad imbalanceNode disk full

   Trigger

   At 05:35 UTC on 4 February the transaction ID counter of shard4, the busiest Mandrill PostgreSQL shard, reached its upper limit and PostgreSQL shut the database down to protect against wraparound.

   Root cause

   Capacity exhaustion

   Component

   DatabasePostgreSQL vacuum

   Contributing factor

   Ignored early signalNo alert on limitSignal buried in noiseRetry stormMonitoring blind spot

   Detection

   The first exception was logged overnight but was buried by unrelated exceptions in the centralized error reporting. Mandrill engineers opened an investigation at 11:58 UTC and, while looking for the source of the rising load on shard4, found the wraparound at 12:56 UTC.

   Remediation

   • Put shard4 into standalone mode and started VACUUM at 14:16 UTC on 4 February
   • Stopped and restarted the vacuum with settings that made it run faster
   • Started a dump and restore of the database without the Search and Url tables while the vacuum ran
   • Replaced storage volumes on the Mandrill app servers whose disks were filling with queued jobs and logs
   • Moved the job-queue tables from shard4 to other live shards
   • Truncated the Search and Url tables on the locked database at 19:12 UTC on 5 February, after which the vacuum finished in about an hour
   • Started sending from the backlog at 21:36 UTC
   • Added alerting for XID wraparound by 22:40 UTC

   Lessons

   Mailchimp named the factors that slowed its response: knowledge of and access to Mandrill systems and monitoring rested with a few people, Mandrill logging needed a locally writable disk, logs of malformed API requests were hard to separate from real application errors, job health was hard to see, and the disk space problem made partial deployments for triage risky. What helped was trained incident commanders with clear roles (incident commanders in 3-hour rotations, scribes, technical liaisons and points of contact between teams), periodic summaries with a list of recent and open decisions, and taking time in a video meeting to reassess, which led to the truncation. Running in the cloud let the team expand storage quickly, and the team was lucky that the largest table holding up the vacuum held data it could delete.

   View case

By root cause4