IL-0556 Storj · 2024-05-20database

A transaction left open during database maintenance stopped uploads and then downloads on Storj's US1 satellite

On 20 May 2024 a routine, manual database maintenance procedure began on the Storj US1 satellite at 14:55 UTC, and a transaction was inadvertently left open while other work went on. An open transaction holds locks on the data it touched, so other operations cannot read or change that data until it is committed or rolled back. Writes began to fail at 14:56, and as the locks stayed in place reads failed as well, disrupting the US1 service. The on-call team was paged at 14:59, closed the open transaction at 15:22, and operations were restored at 15:23. The AP1 and EU1 satellites were not affected.

Severity: –
Scope: Storj US1 satellite storage and retrieval (US1 API, Linksharing, Gateway, Select): uploads failed first, then downloads
Users affected: Customers and applications using the US1 satellite
Region: US1
Time to detect: 3m
Time to resolve: 27m

Symptoms

At 14:56 UTC on 20 May 2024, write operations on the Storj US1 satellite began to fail, so uploads stopped working, and downloads failed after them. The on-call team was paged at 14:59 UTC, while the AP1 and EU1 satellites kept working normally.

Symptom Lock contention or deadlock Service outage

Root cause

Root cause Operator error

Component Database

Contributing factor –

Trigger: During routine, manual database maintenance on the US1 satellite that began at 14:55 UTC, a transaction was left open by mistake while other work was being done.

Detection

The on-call team received a page at 14:59 UTC, three minutes after writes started failing.

Remediation

Found and closed the open transaction at 15:22 UTC, after which the satellite recovered

Action items

Review and update database maintenance procedures and training so that every transaction is closed before moving on to other tasks
Add monitoring and alerts for open transactions that last longer than a set duration
Look for further process and system improvements, including tools that reduce or remove the need for manual maintenance

Lessons

–

Sources

https://status.storj.io/incidents/brtpk2c22gdz · official-status · archived copy

Related cases

IL-0577 Ficilcom's production Neon PostgreSQL database was reset by prisma migrate reset meant for the local databaseFicilcom · 2026
same root cause · shared Database
IL-0576 Every dataset in CareNet's production BigQuery data warehouse was deleted, and UNDROP SCHEMA could not restore themCareNet · 2025
same root cause · shared Database
IL-0242 Both PostgreSQL servers behind matrix.org were lost during a disk expansion, and the homeserver was down for 24 hoursMatrix.org Foundation · 2025
same root cause · shared Database
IL-0582 Jobs piled up in Kanmu's PostgreSQL-based qg job queue on Aurora as dequeue slowed and dead tuples kept growingKanmu · 2024
same root cause · shared Database
IL-0531 Azure Database for MySQL Single Server and MariaDB in West US rejected every new connection for a batch of serversMicrosoft Azure · 2024
same root cause · shared Database
IL-0565 Cookie Run: Kingdom's CockroachDB cluster failed when 16 of 24 nodes lost their filesystems, causing a 36-hour outageDevsisters · 2022
same root cause · shared Database
IL-0207 A routine task run from an engineer's local environment switched to production and deleted GitLab.com's Consul release, so Patroni lost Consul and GitLab.com returned 5xx errors to all trafficGitLab · 2022
same root cause · shared Database
IL-0332 A test run against the production database truncated every table and travis-ci.com was down for about 5.5 hoursTravis CI · 2018
same root cause · shared Database
IL-0431 During Mackerel's move to AWS, PostgreSQL failed over to a replica that had stopped replicating, losing an hour of dataHatena · 2017
same root cause · shared Database
IL-0506 UC Berkeley DataHub logins failed and students got empty home directories after the JupyterHub database was overwrittenUC Berkeley · 2017
same root cause · shared Database
IL-0218 GitLab.com's primary PostgreSQL data was wiped by mistake and, with no working backup, the site was down about 18 hoursGitLab · 2017
same root cause · shared Database
IL-0573 A developer's mistake dropped nine main tables of the Baemin billing MariaDB database and payments briefly stoppedWoowa Brothers · 2016
same root cause · shared Database
IL-0319 Dropping databases on the MySQL replica mysql2 also dropped them on mysql1, as the pair actually ran master-masterOregon State University Open Source Lab · 2015
same root cause · shared Database
IL-0337 EBS volumes in one US East Availability Zone got stuck after a network change cut EBS nodes off from their replicasAmazon Web Services · 2011
same root cause · shared Database

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Database among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0556
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0556/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.