Incident Library
Applied filters Root causeunknownComponentDatabaseSymptomService outageClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases3 cases

1 to 3 of 3

  1. IL-0545 Wasabi API calls in all regions returned HTTP 5XX when the message queue stopped spreading requests over database nodes
    Wasabi2024
    Symptoms

    From 00:17 UTC on 30 November 2024, client connections to Wasabi Cloud Storage and the Web Console failed in every storage region, and all API calls returned HTTP 5XX errors.

    Service outageLoad imbalance
    Root cause
    unknown
    Component
    Message brokerDatabase
    Contributing factor
    –
    Remediation
    • Configured internal servers by hand to route requests across multiple database instances, which restored service
    • After service was restored, worked on the root cause by recovering the internal messaging queue and resuming its automated handling of client requests
    View case
  2. IL-0335 A bugcheck in the primary SQL Server put the Stack Exchange network into read-only mode and then offline for 12 minutes
    Stack Exchange2017
    Symptoms

    On January 24, 2017 at 17:53 UTC the Stack Exchange network, including Stack Overflow, went into read-only mode for about 5 minutes and then went offline for 12 minutes. It took the team 2 minutes to notice the problem, 5 minutes to locate its source and 10 minutes to restore service.

    Writes rejectedService outageFailover failureData loss
    Trigger

    At 17:53 UTC the primary SQL Server, NY-SQL02, triggered a bugcheck in the SQL Server process.

    Root cause
    unknown
    Component
    Database
    Contributing factor
    Single point of failure
    Remediation
    • Restarted the SQL service on NY-SQL02, which brought the network back online in read-only mode
    • Put the sites back into read-write mode after a sanity check of SQL health
    • Brought the network online without waiting for the memory dump to complete
    • Took NY-SQL02 out of production for testing, including a memory test, and ran the normal load on NY-SQL01
    • Brought the SQL cluster up to 2016 SP1 CU1
    View case
  3. IL-0393 Load on the BBC programme metadata database surged and metadata requests failed, leaving BBC iPlayer down over a weekend
    BBC2014
    Symptoms

    At 9.30 on Saturday 19 July 2014, the load on the database of the BBC system that provides programme and clip metadata rose sharply, and many requests for metadata to its application servers started to fail. BBC iPlayer, BBC iPlayer Radio and audio and video playback on other parts of bbc.co.uk were affected.

    Database overloadHigh error rateService outage
    Root cause
    unknown
    Component
    Database
    Contributing factor
    No cachingOverlapping failures
    Remediation
    • Restored the failed pool of the caching layer first
    • Switched the BBC homepage to its emergency mode until the cache was stable
    • Did significant remedial work on the metadata system from Saturday afternoon into the evening
    • Ran the service in a reduced state through the weekend and did the full restoration on Monday morning
    Action items
    • Complete the forensics on the root causes and put measures in place to reduce the chance of similar interruptions
    View case
Check your cluster for this

These 3 cases involve Database and Message broker. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

unknown3 cases

Component
Database3Message broker1
Contributing factor
No caching1Overlapping failures1Single point of failure1
Organisation
BBC1Stack Exchange1Wasabi1
  1. IL-0545 Wasabi API calls in all regions returned HTTP 5XX when the message queue stopped spreading requests over database nodes
    Wasabi2024
  2. IL-0335 A bugcheck in the primary SQL Server put the Stack Exchange network into read-only mode and then offline for 12 minutes
    Stack Exchange2017
  3. IL-0393 Load on the BBC programme metadata database surged and metadata requests failed, leaving BBC iPlayer down over a weekend
    BBC2014