Cases3 cases

1 to 3 of 3

1. IL-0551 Bitbucket Cloud's overloaded primary database failed over, and the website, API and Pipelines were slow for almost a day

   Atlassian2025

   Symptoms

   At 15:26 UTC on 8 May 2025, the Bitbucket Cloud website and API became slow because the primary database was overloaded. Atlassian's real-time monitoring detected the problem at once.

   Database overloadHigh latencyHigh error rate

   Trigger

   A high-scale backfill job run by an internal Atlassian service sent a large volume of calls to certain Bitbucket API endpoints, which ran expensive queries against the primary database.

   Root cause

   Capacity exhaustion

   Component

   Database

   Contributing factor

   Inefficient queryRetry storm

   Detection

   Atlassian's real-time monitoring detected the incident as soon as it started.

   Remediation

   • Stopped the high-intensity backfill job
   • Primary database failed over automatically, and Bitbucket services recovered within 15 minutes
   • Applied several measures after the failover to shed database load

   Action items

   • Route more reads to read replicas instead of the write primary database
   • Adjust rate limits for internal API endpoints that run resource-intensive database operations
   • Optimise database queries so they run more efficiently
   • Tune the retry policies of downstream services

   View case
2. IL-0550 Bitbucket Cloud's database slowed until connection pools saturated and requests timed out, leaving Pipelines pending

   Atlassian2024

   Symptoms

   From 07:22 UTC on 22 February 2024, the bitbucket.org website and the Bitbucket Cloud APIs became slow, and Bitbucket Pipelines runs were badly delayed. Git requests over HTTPS and SSH kept working.

   High latencyConnection pool exhaustionIntermittent timeoutsThroughput degradation

   Root cause

   Capacity exhaustion

   Component

   DatabasePostgreSQL vacuum

   Contributing factor

   No alert on limit

   Remediation

   • Added resources to Bitbucket Pipelines to clear the backlog of jobs, which was fully cleared by 17:30 UTC

   Action items

   • Reconfigure the vacuuming threshold for database tables with high write activity
   • Adjust alert thresholds to catch this behaviour earlier
   • Tune autoscaling and load shedding for the Pipelines services and increase build runner capacity

   View case
3. IL-0549 Media, attachments, Git LFS and build artifacts failed for Atlassian in us-west-1 when an old network stack was removed

   Atlassian2023

   Symptoms

   From 02:36 UTC on 15 May 2023, Atlassian products with services hosted in the us-west-1 region could not store or retrieve data assets such as media, attachments and build artifacts. Automated monitoring detected it within three minutes, and the status page reported intermittent errors.

   High error rateStale endpoint routing

   Trigger

   An internal service was migrated to a new network segment as part of an initiative to harden the partitions between network segments, and the old network stack was then removed.

   Root cause

   Configuration change

   Component

   Authoritative DNS servers and records

   Contributing factor

   Unverified assumptionUnrepresentative staging

   Detection

   Automated monitoring detected the incident within three minutes.

   Remediation

   • Rolled back the network migration change, returning Atlassian systems to a known good state

   Action items

   • Review the systems that decommission service stacks and check that customer traffic is no longer served before a stack is decommissioned
   • Add steps to the service network migration process that identify associated DNS records needing attention

   Lessons

   Atlassian said the migration used blue/green deployment to limit this kind of risk, but adequate verification was not run before the old stack was decommissioned, and moving services between network segments is rare and hard to replicate in test.

   View case

By root cause2