Incident Library
Applied filters Root causeunknownComponentDatabaseSymptomDatabase overloadClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases3 cases

1 to 3 of 3

  1. IL-0441 A ZOZO SQL Server database stalled and applications hit timeout storms whenever batch executions peaked in sales
    ZOZO2021
    Symptoms

    From around mid-2020, a SQL Server database behind one ZOZO feature made the applications throw many timeout errors whenever batch executions per second passed a certain threshold.

    Intermittent timeoutsDatabase overloadLock contention or deadlock
    Trigger

    Traffic increases during sales and lucky-bag launches pushed batch executions per second on the database above a threshold.

    Root cause
    unknown
    Component
    Database
    Contributing factor
    –
    Remediation
    • Fixed the plan guides of the four stored procedures whose compilation waits caused blocking
    • Disabled automatic statistics updates on the affected tables during periods of expected high traffic
    • Enabled automatic statistics updates again once traffic calmed down
    • Compressed the large indexes held in the data cache to use memory more efficiently
    Lessons

    When the root cause of a failure cannot be found, choosing among the things observed during the failure those that can be removed or reduced, and acting on them, can still create a state in which the failure no longer happens.

    View case
  2. IL-0387 Mailgun MongoDB secondaries were overloaded by rising connections, and a rollback then moved reads onto the primary too
    Mailgun2017
    Symptoms

    On 12 January 2017 at 14:09 UTC, Mailgun engineers began responding to alerts for the click and open tracking services and the primary MongoDB clusters in one production region. Within ten minutes the engineers found that the MongoDB secondary servers, which serve most reads, were under heavy load and slow to answer, which also caused the tracking errors. The status page reported intermittent timeouts on the Control Panel, API, SMTP and tracking links.

    Database overloadIntermittent timeoutsHigh error rate
    Trigger

    A gradual increase in connections to the MongoDB secondary servers, which coincided with several MongoDB configuration changes made the day before. A rollback of those changes then also introduced, by mistake, a change that moved the API and SMTP services from the secondaries to the primary.

    Root cause
    unknown
    Component
    Database
    Contributing factor
    Harmful manual actionSingle point of failure
    Detection

    Alerts for the click and open tracking services and the primary MongoDB clusters at 14:09 UTC.

    Remediation
    • Rolled back the previous day's MongoDB configuration changes through several pull requests from 15:15 UTC
    • Resized the MongoDB servers to handle more connections
    • Deployed connection limits at the MongoDB server level
    • Limited the number of connections the services could open to the MongoDB primary
    • Redeployed the configuration that makes services prefer secondary servers for reads
    Action items
    • Run the tracking infrastructure across two independent regions
    • Limit the number of connections every service can open to the database
    • Permanently increase the size of the primary Mongo cluster and roll the change out to the other data centers
    • Let click redirection keep working when downstream services fail
    View case
  3. IL-0393 Load on the BBC programme metadata database surged and metadata requests failed, leaving BBC iPlayer down over a weekend
    BBC2014
    Symptoms

    At 9.30 on Saturday 19 July 2014, the load on the database of the BBC system that provides programme and clip metadata rose sharply, and many requests for metadata to its application servers started to fail. BBC iPlayer, BBC iPlayer Radio and audio and video playback on other parts of bbc.co.uk were affected.

    Database overloadHigh error rateService outage
    Root cause
    unknown
    Component
    Database
    Contributing factor
    No cachingOverlapping failures
    Remediation
    • Restored the failed pool of the caching layer first
    • Switched the BBC homepage to its emergency mode until the cache was stable
    • Did significant remedial work on the metadata system from Saturday afternoon into the evening
    • Ran the service in a reduced state through the weekend and did the full restoration on Monday morning
    Action items
    • Complete the forensics on the root causes and put measures in place to reduce the chance of similar interruptions
    View case
Check your cluster for this

These 3 cases involve Database. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

unknown3 cases

Component
Database3
Contributing factor
Harmful manual action1No caching1Overlapping failures1Single point of failure1
Organisation
BBC1Mailgun1ZOZO1
  1. IL-0441 A ZOZO SQL Server database stalled and applications hit timeout storms whenever batch executions peaked in sales
    ZOZO2021
  2. IL-0387 Mailgun MongoDB secondaries were overloaded by rising connections, and a rollback then moved reads onto the primary too
    Mailgun2017
  3. IL-0393 Load on the BBC programme metadata database surged and metadata requests failed, leaving BBC iPlayer down over a weekend
    BBC2014