Incident Library
Applied filters SymptomHigh latencyOrganisationAtlassianClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0551 Bitbucket Cloud's overloaded primary database failed over, and the website, API and Pipelines were slow for almost a day
    Atlassian2025
    Symptoms

    At 15:26 UTC on 8 May 2025, the Bitbucket Cloud website and API became slow because the primary database was overloaded. Atlassian's real-time monitoring detected the problem at once.

    Database overloadHigh latencyHigh error rate
    Trigger

    A high-scale backfill job run by an internal Atlassian service sent a large volume of calls to certain Bitbucket API endpoints, which ran expensive queries against the primary database.

    Root cause
    Capacity exhaustion
    Component
    Database
    Contributing factor
    Inefficient queryRetry storm
    Detection

    Atlassian's real-time monitoring detected the incident as soon as it started.

    Remediation
    • Stopped the high-intensity backfill job
    • Primary database failed over automatically, and Bitbucket services recovered within 15 minutes
    • Applied several measures after the failover to shed database load
    Action items
    • Route more reads to read replicas instead of the write primary database
    • Adjust rate limits for internal API endpoints that run resource-intensive database operations
    • Optimise database queries so they run more efficiently
    • Tune the retry policies of downstream services
    View case
  2. IL-0550 Bitbucket Cloud's database slowed until connection pools saturated and requests timed out, leaving Pipelines pending
    Atlassian2024
    Symptoms

    From 07:22 UTC on 22 February 2024, the bitbucket.org website and the Bitbucket Cloud APIs became slow, and Bitbucket Pipelines runs were badly delayed. Git requests over HTTPS and SSH kept working.

    High latencyConnection pool exhaustionIntermittent timeoutsThroughput degradation
    Root cause
    Capacity exhaustion
    Component
    DatabasePostgreSQL vacuum
    Contributing factor
    No alert on limit
    Remediation
    • Added resources to Bitbucket Pipelines to clear the backlog of jobs, which was fully cleared by 17:30 UTC
    Action items
    • Reconfigure the vacuuming threshold for database tables with high write activity
    • Adjust alert thresholds to catch this behaviour earlier
    • Tune autoscaling and load shedding for the Pipelines services and increase build runner capacity
    View case
Check your cluster for this

These 2 cases involve Database and PostgreSQL vacuum. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion2 cases

Component
Database2PostgreSQL vacuum1
Contributing factor
Inefficient query1No alert on limit1Retry storm1
Organisation
Atlassian2
  1. IL-0551 Bitbucket Cloud's overloaded primary database failed over, and the website, API and Pipelines were slow for almost a day
    Atlassian2025
  2. IL-0550 Bitbucket Cloud's database slowed until connection pools saturated and requests timed out, leaving Pipelines pending
    Atlassian2024