Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0551 Bitbucket Cloud's overloaded primary database failed over, and the website, API and Pipelines were slow for almost a day
    Atlassian2025
    Symptoms

    At 15:26 UTC on 8 May 2025, the Bitbucket Cloud website and API became slow because the primary database was overloaded. Atlassian's real-time monitoring detected the problem at once.

    Database overloadHigh latencyHigh error rate
    Trigger

    A high-scale backfill job run by an internal Atlassian service sent a large volume of calls to certain Bitbucket API endpoints, which ran expensive queries against the primary database.

    Root cause
    Capacity exhaustion
    Component
    Database
    Contributing factor
    Inefficient queryRetry storm
    Detection

    Atlassian's real-time monitoring detected the incident as soon as it started.

    Remediation
    • Stopped the high-intensity backfill job
    • Primary database failed over automatically, and Bitbucket services recovered within 15 minutes
    • Applied several measures after the failover to shed database load
    Action items
    • Route more reads to read replicas instead of the write primary database
    • Adjust rate limits for internal API endpoints that run resource-intensive database operations
    • Optimise database queries so they run more efficiently
    • Tune the retry policies of downstream services
    View case
Check your cluster for this

This case involves Database. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion1 case

Component
Database
Contributing factor
Inefficient queryRetry storm
Organisation
Atlassian
  1. IL-0551 Bitbucket Cloud's overloaded primary database failed over, and the website, API and Pipelines were slow for almost a day
    Atlassian2025