IL-0545 Wasabi · 2024-11-30database

Wasabi API calls in all regions returned HTTP 5XX when the message queue stopped spreading requests over database nodes

Wasabi's internal messaging queue service takes client requests and routes them to the global database cluster. On 30 November 2024 an error in that queue service stopped it from routing client requests across all of the database nodes, and from 00:17 UTC to about 03:00 UTC client requests to Wasabi Cloud Storage and the Web Console failed in all storage regions with HTTP 5XX errors. The engineering and operations teams restored service by configuring internal servers by hand to route requests across several database instances, and then worked on the root cause by recovering the internal messaging queue and resuming its automated handling of client requests. Errors in us-east-1 lasted longer than in the other regions, and the status page reported all regions fully operational at 04:45 UTC.

Severity: –
Scope: Client connections to Wasabi Cloud Storage and the Web Console, with all API calls returning HTTP 5XX errors; the status page also lists the Wasabi Account Control API, the Wasabi Account Control Manager Console and the Wasabi Management Console
Users affected: –
Region: All Wasabi storage regions; us-east-1 (Ashburn) recovered last
Time to detect: –
Time to resolve: 2h 43m

Symptoms

From 00:17 UTC on 30 November 2024, client connections to Wasabi Cloud Storage and the Web Console failed in every storage region, and all API calls returned HTTP 5XX errors.

The status page first described the problem as network errors across all regions, with both the Console and S3 returning errors. Later the status page said the issue was isolated and resolution was underway, and that it expected every region except us-east-1 to recover, while some requests in Ashburn still got error responses. At 04:45 UTC it reported the system fully operational in all regions.

Symptom Service outage Load imbalance

Root cause

Root cause unknown

Component Message broker Database

Contributing factor –

Trigger: –

Detection

–

Remediation

Configured internal servers by hand to route requests across multiple database instances, which restored service
After service was restored, worked on the root cause by recovering the internal messaging queue and resuming its automated handling of client requests

Action items

–

Lessons

–

Sources

https://status.wasabi.com/incidents/dcb131s9f3sy · official-status · archived copy

Related cases

IL-0441 A ZOZO SQL Server database stalled and applications hit timeout storms whenever batch executions peaked in salesZOZO · 2021
same root cause · shared Database
IL-0335 A bugcheck in the primary SQL Server put the Stack Exchange network into read-only mode and then offline for 12 minutesStack Exchange · 2017
same root cause · shared Database
IL-0387 Mailgun MongoDB secondaries were overloaded by rising connections, and a rollback then moved reads onto the primary tooMailgun · 2017
same root cause · shared Database
IL-0393 Load on the BBC programme metadata database surged and metadata requests failed, leaving BBC iPlayer down over a weekendBBC · 2014
same root cause · shared Database

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Message broker, Database among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0545
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0545/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.