Cases6 cases

1 to 5 of 6

1. IL-0287 One overloaded shard of MongoDB's internal monitoring database delayed Atlas metrics, alerts and cluster operations

   MongoDB2025

   Symptoms

   On 7 October 2025 MongoDB reported a delay in the metric ingestion pipeline of MongoDB Atlas and Cloud Manager. Metrics in Atlas dashboards arrived late or went missing, Host Down alerts did not fire, and Atlas cluster management operations such as provisioning and resizing were delayed, while customer clusters themselves stayed healthy and available.

   Monitoring outageDatabase overloadLoad imbalance

   Trigger

   The rollout of an updated internal software version caused a temporary spike in resource demand on the internal database cluster.

   Root cause

   Capacity exhaustion

   Component

   Database

   Contributing factor

   Inefficient queryNo headroom

   Remediation

   • Took corrective actions to stabilise the system and monitored their effect
   • Implemented a fix and monitored the results until the incident was resolved

   Action items

   • Shard the collections with concentrated load to spread traffic over more nodes
   • Optimise inefficient queries
   • Provision additional infrastructure capacity for higher traffic
   • Refine capacity planning to anticipate future load spikes, using metrics from this event
   • Redesign software rollouts to account for the predictable rise in resource demand during a rollout
   • Keep monitoring how the sharding and query changes resolve the hotspots
   • Expand the detection of elevated load conditions for faster anomaly detection and response

   View case
2. IL-0551 Bitbucket Cloud's overloaded primary database failed over, and the website, API and Pipelines were slow for almost a day

   Atlassian2025

   Symptoms

   At 15:26 UTC on 8 May 2025, the Bitbucket Cloud website and API became slow because the primary database was overloaded. Atlassian's real-time monitoring detected the problem at once.

   Database overloadHigh latencyHigh error rate

   Trigger

   A high-scale backfill job run by an internal Atlassian service sent a large volume of calls to certain Bitbucket API endpoints, which ran expensive queries against the primary database.

   Root cause

   Capacity exhaustion

   Component

   Database

   Contributing factor

   Inefficient queryRetry storm

   Detection

   Atlassian's real-time monitoring detected the incident as soon as it started.

   Remediation

   • Stopped the high-intensity backfill job
   • Primary database failed over automatically, and Bitbucket services recovered within 15 minutes
   • Applied several measures after the failover to shed database load

   Action items

   • Route more reads to read replicas instead of the write primary database
   • Adjust rate limits for internal API endpoints that run resource-intensive database operations
   • Optimise database queries so they run more efficiently
   • Tune the retry policies of downstream services

   View case
3. IL-0447 An every RDS for MySQL instance slowed badly when its gp2 EBS BurstBalance ran out and IOPS fell to the baseline

   every2024

   Symptoms

   The BurstBalance of an RDS instance began to fall rapidly and finally ran out, and the database's performance dropped sharply. The instance ran MySQL 5.7.44 on db.t3.medium with general purpose SSD (gp2) storage of 30GiB.

   Database overloadThroughput degradation

   Trigger

   A complex SELECT query began writing large amounts of data to temporary tables, which raised WriteIOPS and used up the burst credits.

   Root cause

   Capacity exhaustion

   Component

   DatabaseCloud block storage

   Contributing factor

   Inefficient query

   Action items

   • Consider tuning the query by reviewing indexes and rewriting it
   • Consider moving the storage from gp2 to gp3 or increasing its size to raise the baseline IOPS
   • Consider adjusting tmp_table_size and max_heap_table_size
   • Consider moving to a more powerful instance type
   • Consider monitoring BurstBalance with CloudWatch alerts and adding external monitoring such as Pingdom

   Lessons

   A complex query can have a large effect on I/O, so database design and queries need continuous improvement, and query performance needs regular monitoring.

   View case
4. IL-0302 Peak load on GitHub's mysql1 MySQL cluster maxed out its database proxy's connections and stopped writes in March 2022

   GitHub2022

   Symptoms

   On March 16, 2022 at 14:09 UTC, write operations across GitHub stopped working, including git operations, webhooks, pull requests, API requests, issues, GitHub Packages, GitHub Codespaces, GitHub Actions and GitHub Pages. Under peak-hour load, the database proxy in front of the shared mysql1 MySQL database had reached its maximum number of connections.

   Writes rejectedDatabase overloadHigh error rateFailover failure

   Trigger

   Increased load on mysql1 during peak hours, combined with poor query performance in specific circumstances, pushed the database proxy to its maximum number of connections.

   Root cause

   Capacity exhaustion

   Component

   DatabaseDatabase proxy and connection pooler

   Contributing factor

   Inefficient queryChange during peak hoursHarmful manual actionExcessive blast radius

   Remediation

   • Failed over mysql1 to a healthy replica on March 16
   • Reset the application connections to the new primary on March 17
   • Added an index that fixed the main query performance problem
   • Performed a primary failover on March 22
   • Performed a primary failover on March 23
   • Throttled webhook traffic to reduce load

   Action items

   • Keep throttling webhook traffic at peak load times
   • Audit the load patterns of mysql1 during peak hours and make performance fixes based on the audits
   • Move traffic to other databases to reduce load and speed up failover
   • Review change management, particularly monitoring and changes during high load in production
   • Shard the databases and scale up hardware

   View case
5. IL-0212 Many Slack users could not connect because a Vitess keyspace for channel membership was overloaded by cache-miss queries

   Slack2022

   Symptoms

   Just after 6 a.m. Pacific Time on 22 February 2022, user tickets about problems connecting to Slack, problems reported by internal users, and pages and alerts to several engineering teams all arrived at almost the same moment. Many users could not connect: the Slack client could not finish booting, the step that fetches channel lists, preferences and recent conversations, and showed an error page.

   Database overloadHigh error rateHigh latency

   Trigger

   A percentage-based rollout step upgraded the Consul agent on 25% of the fleet, followed by a slow sequential restart of the agents. Each restart on a Memcached node removed it from the Consul service catalog, so Mcrib, the cache-tier control plane, replaced it with an empty spare or flushed it when it rejoined, and the cache hit rate dropped as Slack reached its daily peak traffic.

   Root cause

   Capacity exhaustion

   Component

   DatabaseIn-memory cacheService discovery

   Contributing factor

   Dependence on a warm cacheInefficient queryAutomation amplificationRetry stormChange during peak hours

   Detection

   User tickets about connection problems, reports from internal users and pages and alerts to several engineering teams arrived almost at the same time, just after 6 a.m. Pacific Time.

   Remediation

   • Paused the Consul agent restart operation
   • Throttled client boot requests to reduce load on the database tier
   • Cut the client boot rate limit again after a too-large increase and then raised it in smaller steps
   • Changed the GDM membership query to read from Vitess only the data missing from Memcached
   • Let the query read from replicas as well as Vitess primaries
   • Raised the client boot rate limit slowly back to normal traffic levels as the caches refilled

   Action items

   • Change the process for Consul rollouts so upgrades do not trigger this failure mode again
   • Change the Mcrib control loop so a brief network outage or partition of cache nodes does not set off the same cascading failure
   • Make the GDM membership query read from a table sharded by channel
   • Check the other cache-fronted database queries for high-volume scatter queries
   • Explore longer-term ways to make the caching tier more resilient

   Lessons

   Slack describes performance bottlenecks, a cascading failure, dependence on a warm cache and a change as the trigger. Once the system had passed the tipping point, lower cache hit rates caused high database load that in turn kept the cache from filling, and a system in that state stays there until an outside intervention, a large cut in load or an increase in capacity, changes it. Mcrib contributed not by being wrong but by being faster than the old Consul lock scheme at replacing Memcached nodes, which increased churn in the cache tier. Client retries still added load despite backoff and jitter. A brief network partition of the cache nodes could set off the same failure, and the interaction between Consul and the caching architecture is now more widely understood.

   View case

By root cause1