Incident Library
Applied filters Root causeCapacity exhaustionComponentIn-memory cacheOrganisationWasabiClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0544 Wasabi's database cache stopped responding when its queueing service filled up, and S3 failed in all regions
    Wasabi2024
    Symptoms

    From 10:32 UTC on 6 August 2024, Wasabi's S3 service and user services failed in every region: users had trouble logging into the Web Console and requests to Wasabi regions returned errors.

    Service outageFailover failureIntermittent timeouts
    Trigger

    At 10:32 UTC on 6 August, Wasabi's queueing service reached full capacity, and that made the database cache unresponsive.

    Root cause
    Capacity exhaustion
    Component
    Message brokerIn-memory cacheDatabase
    Contributing factor
    –
    Remediation
    • Restarted the primary database to clear stale connections and cleared the queueing service's queue at the same time, which did not fully restore the database
    • Promoted the secondary database instance to primary, after which S3 was fully operational at 11:20 UTC
    • Restarted the database again between 13:17 and 13:23 UTC to fully incorporate the queueing service library
    • Promoted the original primary database back to production on 7 August, which stopped the user service timeouts
    • Recreated and redeployed the automation cluster that the automation service could not see, which fully restored S3
    View case
Check your cluster for this

This case involves In-memory cache and Database. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion1 case

Component
In-memory cacheDatabaseMessage broker
Contributing factor
–
Organisation
Wasabi
  1. IL-0544 Wasabi's database cache stopped responding when its queueing service filled up, and S3 failed in all regions
    Wasabi2024