Cases2 cases

1 to 2 of 2

1. IL-0557 Database transaction locks piled up on Storj's US1 satellite, and uploads and downloads failed with timeouts for 2 hours

   Storj2025

   Symptoms

   From 21:04 UTC on 7 August 2025, database transaction locks on the Storj US1 satellite started to increase, and upload and download requests began to fail. The on-call team was paged at 21:14 UTC.

   Lock contention or deadlockIntermittent timeoutsConnection pool exhaustionHigh error rate

   Trigger

   An unusually high volume of concurrent uploads to the same objects reached the US1 satellite.

   Root cause

   Capacity exhaustion

   Component

   Database

   Contributing factor

   –

   Detection

   The on-call team was paged at 21:14 UTC, ten minutes after database transaction locks began to rise.

   Remediation

   • Reset the state of connections periodically to keep contention errors from growing in a cascade

   Action items

   • Remove the sources of contention between database transactions with code changes
   • Add monitoring and alerts for increased levels of database locks
   • Look for further process and system improvements, including tools that reduce or remove the need for manual maintenance

   View case
2. IL-0556 A transaction left open during database maintenance stopped uploads and then downloads on Storj's US1 satellite

   Storj2024

   Symptoms

   At 14:56 UTC on 20 May 2024, write operations on the Storj US1 satellite began to fail, so uploads stopped working, and downloads failed after them. The on-call team was paged at 14:59 UTC, while the AP1 and EU1 satellites kept working normally.

   Lock contention or deadlockService outage

   Trigger

   During routine, manual database maintenance on the US1 satellite that began at 14:55 UTC, a transaction was left open by mistake while other work was being done.

   Root cause

   Operator error

   Component

   Database

   Contributing factor

   –

   Detection

   The on-call team received a page at 14:59 UTC, three minutes after writes started failing.

   Remediation

   • Found and closed the open transaction at 15:22 UTC, after which the satellite recovered

   Action items

   • Review and update database maintenance procedures and training so that every transaction is closed before moving on to other tasks
   • Add monitoring and alerts for open transactions that last longer than a set duration
   • Look for further process and system improvements, including tools that reduce or remove the need for manual maintenance

   View case

By root cause2