Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0298 Wikimedia edits failed for 38 minutes when sessionstore Cassandra nodes in both data centres crashed on full disks
    Wikimedia Foundation2025
    Symptoms

    At 02:58 UTC on 31 March 2025, the sessionstore service in both Wikimedia data centres began returning 500 errors and edits failed. MediaWikiEditFailures (session loss) fired as critical at 03:03, and SessionStoreErrorRateHigh paged at 03:04.

    High error rateService outageNode disk fullData loss
    Trigger

    A rise in session writes, made up of session overwrites at a high rate, filled the storage of the sessionstore Cassandra nodes. The review says the rollout of SUL3 correlates strongly with it.

    Root cause
    Capacity exhaustion
    Component
    Database
    Contributing factor
    No alert on limitUnverified assumption
    Detection

    MediaWikiEditFailures (session loss) fired as critical at 03:03 UTC, and SessionStoreErrorRateHigh, a high 5xx rate alert, paged the batphone at 03:04. Users also reported the problem in task T390512.

    Remediation
    • Wiped the sessionstore storage, removing all sessions, and restarted Cassandra
    Action items
    • Page when the disk utilisation trend becomes unreasonable
    • Increase sessionstore storage capacity
    • Add namespacing to sessionstore
    • Add observability of the sessionstore workload
    • Identify the cause of the high rate of storage growth in sessionstore Cassandra
    • Consider an alert, without paging, on the Cassandra error rate
    • Consider paging or alerting when Cassandra nodes enter a down state
    Lessons

    The storage growth before the node failures made their timing predictable, so an alert on high storage utilisation well in advance would have given a chance to act before users were affected, and an alert on Cassandra node failures would have told responders more than an HTTP error rate alert. A cluster seen as heavily over-provisioned can still grow quickly and unsustainably under an unusual workload.

    View case
Check your cluster for this

This case involves Database. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion1 case

Component
Database
Contributing factor
Unverified assumptionNo alert on limit
Organisation
Wikimedia Foundation
  1. IL-0298 Wikimedia edits failed for 38 minutes when sessionstore Cassandra nodes in both data centres crashed on full disks
    Wikimedia Foundation2025