Cases2 cases

1 to 2 of 2

1. IL-0287 One overloaded shard of MongoDB's internal monitoring database delayed Atlas metrics, alerts and cluster operations

   MongoDB2025

   Symptoms

   On 7 October 2025 MongoDB reported a delay in the metric ingestion pipeline of MongoDB Atlas and Cloud Manager. Metrics in Atlas dashboards arrived late or went missing, Host Down alerts did not fire, and Atlas cluster management operations such as provisioning and resizing were delayed, while customer clusters themselves stayed healthy and available.

   Monitoring outageDatabase overloadLoad imbalance

   Trigger

   The rollout of an updated internal software version caused a temporary spike in resource demand on the internal database cluster.

   Root cause

   Capacity exhaustion

   Component

   Database

   Contributing factor

   Inefficient queryNo headroom

   Remediation

   • Took corrective actions to stabilise the system and monitored their effect
   • Implemented a fix and monitored the results until the incident was resolved

   Action items

   • Shard the collections with concentrated load to spread traffic over more nodes
   • Optimise inefficient queries
   • Provision additional infrastructure capacity for higher traffic
   • Refine capacity planning to anticipate future load spikes, using metrics from this event
   • Redesign software rollouts to account for the predictable rise in resource demand during a rollout
   • Keep monitoring how the sharding and query changes resolve the hotspots
   • Expand the detection of elevated load conditions for faster anomaly detection and response

   View case
2. IL-0375 MongoDB Atlas cluster changes and alerts lagged for hours after a control plane metadata migration congested a database

   MongoDB2024

   Symptoms

   From 18:24 UTC on 3 June 2024, MongoDB Atlas cluster configuration changes in all commercial regions ran with delays of a few minutes to several hours. The delays hit changes requested through the API and changes from automated triggers such as auto-scaling and node replacement.

   Control plane lagMonitoring outageDatabase overload

   Trigger

   During planned internal maintenance, the metadata of an internal workflow management system that backs the Atlas control plane was migrated to a target database cluster, and traffic was redirected to that target.

   Root cause

   Capacity exhaustion

   Component

   Database

   Contributing factor

   Unrepresentative stagingIneffective rollback

   Remediation

   • Rolled back the metadata migration, reconciling data between the source and target databases
   • Executed recovery procedures, completing them at 00:09 UTC on 4 June, and kept monitoring closely

   Action items

   • Design a new metadata migration strategy that can roll back within 15 minutes of detecting a problem
   • Hold similar maintenance activities until the new procedure is implemented and tested

   View case

By root cause1