Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0545 Wasabi API calls in all regions returned HTTP 5XX when the message queue stopped spreading requests over database nodes
    Wasabi2024
    Symptoms

    From 00:17 UTC on 30 November 2024, client connections to Wasabi Cloud Storage and the Web Console failed in every storage region, and all API calls returned HTTP 5XX errors.

    Service outageLoad imbalance
    Root cause
    unknown
    Component
    Message brokerDatabase
    Contributing factor
    –
    Remediation
    • Configured internal servers by hand to route requests across multiple database instances, which restored service
    • After service was restored, worked on the root cause by recovering the internal messaging queue and resuming its automated handling of client requests
    View case
  2. IL-0544 Wasabi's database cache stopped responding when its queueing service filled up, and S3 failed in all regions
    Wasabi2024
    Symptoms

    From 10:32 UTC on 6 August 2024, Wasabi's S3 service and user services failed in every region: users had trouble logging into the Web Console and requests to Wasabi regions returned errors.

    Service outageFailover failureIntermittent timeouts
    Trigger

    At 10:32 UTC on 6 August, Wasabi's queueing service reached full capacity, and that made the database cache unresponsive.

    Root cause
    Capacity exhaustion
    Component
    Message brokerIn-memory cacheDatabase
    Contributing factor
    –
    Remediation
    • Restarted the primary database to clear stale connections and cleared the queueing service's queue at the same time, which did not fully restore the database
    • Promoted the secondary database instance to primary, after which S3 was fully operational at 11:20 UTC
    • Restarted the database again between 13:17 and 13:23 UTC to fully incorporate the queueing service library
    • Promoted the original primary database back to production on 7 August, which stopped the user service timeouts
    • Recreated and redeployed the automation cluster that the automation service could not see, which fully restored S3
    View case
Check your cluster for this

These 2 cases involve Database and Message broker. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause2

unknown1 case

Component
DatabaseMessage broker
Contributing factor
–
Organisation
Wasabi
  1. IL-0545 Wasabi API calls in all regions returned HTTP 5XX when the message queue stopped spreading requests over database nodes
    Wasabi2024

Capacity exhaustion1 case

Component
In-memory cacheDatabaseMessage broker
Contributing factor
–
Organisation
Wasabi
  1. IL-0544 Wasabi's database cache stopped responding when its queueing service filled up, and S3 failed in all regions
    Wasabi2024