Incident Library
Applied filters ComponentDatabaseOrganisationMongoDBClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0287 One overloaded shard of MongoDB's internal monitoring database delayed Atlas metrics, alerts and cluster operations
    MongoDB2025
    Symptoms

    On 7 October 2025 MongoDB reported a delay in the metric ingestion pipeline of MongoDB Atlas and Cloud Manager. Metrics in Atlas dashboards arrived late or went missing, Host Down alerts did not fire, and Atlas cluster management operations such as provisioning and resizing were delayed, while customer clusters themselves stayed healthy and available.

    Monitoring outageDatabase overloadLoad imbalance
    Trigger

    The rollout of an updated internal software version caused a temporary spike in resource demand on the internal database cluster.

    Root cause
    Capacity exhaustion
    Component
    Database
    Contributing factor
    Inefficient queryNo headroom
    Remediation
    • Took corrective actions to stabilise the system and monitored their effect
    • Implemented a fix and monitored the results until the incident was resolved
    Action items
    • Shard the collections with concentrated load to spread traffic over more nodes
    • Optimise inefficient queries
    • Provision additional infrastructure capacity for higher traffic
    • Refine capacity planning to anticipate future load spikes, using metrics from this event
    • Redesign software rollouts to account for the predictable rise in resource demand during a rollout
    • Keep monitoring how the sharding and query changes resolve the hotspots
    • Expand the detection of elevated load conditions for faster anomaly detection and response
    View case
  2. IL-0375 MongoDB Atlas cluster changes and alerts lagged for hours after a control plane metadata migration congested a database
    MongoDB2024
    Symptoms

    From 18:24 UTC on 3 June 2024, MongoDB Atlas cluster configuration changes in all commercial regions ran with delays of a few minutes to several hours. The delays hit changes requested through the API and changes from automated triggers such as auto-scaling and node replacement.

    Control plane lagMonitoring outageDatabase overload
    Trigger

    During planned internal maintenance, the metadata of an internal workflow management system that backs the Atlas control plane was migrated to a target database cluster, and traffic was redirected to that target.

    Root cause
    Capacity exhaustion
    Component
    Database
    Contributing factor
    Unrepresentative stagingIneffective rollback
    Remediation
    • Rolled back the metadata migration, reconciling data between the source and target databases
    • Executed recovery procedures, completing them at 00:09 UTC on 4 June, and kept monitoring closely
    Action items
    • Design a new metadata migration strategy that can roll back within 15 minutes of detecting a problem
    • Hold similar maintenance activities until the new procedure is implemented and tested
    View case
Check your cluster for this

These 2 cases involve Database. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion2 cases

Component
Database2
Contributing factor
Inefficient query1No headroom1Ineffective rollback1Unrepresentative staging1
Organisation
MongoDB2
  1. IL-0287 One overloaded shard of MongoDB's internal monitoring database delayed Atlas metrics, alerts and cluster operations
    MongoDB2025
  2. IL-0375 MongoDB Atlas cluster changes and alerts lagged for hours after a control plane metadata migration congested a database
    MongoDB2024