Incident Library
Applied filters Root causeOperator errorComponentDatabaseSymptomLock contention or deadlockClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0556 A transaction left open during database maintenance stopped uploads and then downloads on Storj's US1 satellite
    Storj2024
    Symptoms

    At 14:56 UTC on 20 May 2024, write operations on the Storj US1 satellite began to fail, so uploads stopped working, and downloads failed after them. The on-call team was paged at 14:59 UTC, while the AP1 and EU1 satellites kept working normally.

    Lock contention or deadlockService outage
    Trigger

    During routine, manual database maintenance on the US1 satellite that began at 14:55 UTC, a transaction was left open by mistake while other work was being done.

    Root cause
    Operator error
    Component
    Database
    Contributing factor
    –
    Detection

    The on-call team received a page at 14:59 UTC, three minutes after writes started failing.

    Remediation
    • Found and closed the open transaction at 15:22 UTC, after which the satellite recovered
    Action items
    • Review and update database maintenance procedures and training so that every transaction is closed before moving on to other tasks
    • Add monitoring and alerts for open transactions that last longer than a set duration
    • Look for further process and system improvements, including tools that reduce or remove the need for manual maintenance
    View case
Check your cluster for this

This case involves Database. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Operator error1 case

Component
Database
Contributing factor
–
Organisation
Storj
  1. IL-0556 A transaction left open during database maintenance stopped uploads and then downloads on Storj's US1 satellite
    Storj2024