Incident Library
Applied filters Root causeCapacity exhaustionOrganisationWikimedia FoundationClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0298 Wikimedia edits failed for 38 minutes when sessionstore Cassandra nodes in both data centres crashed on full disks
    Wikimedia Foundation2025
    Symptoms

    At 02:58 UTC on 31 March 2025, the sessionstore service in both Wikimedia data centres began returning 500 errors and edits failed. MediaWikiEditFailures (session loss) fired as critical at 03:03, and SessionStoreErrorRateHigh paged at 03:04.

    High error rateService outageNode disk fullData loss
    Trigger

    A rise in session writes, made up of session overwrites at a high rate, filled the storage of the sessionstore Cassandra nodes. The review says the rollout of SUL3 correlates strongly with it.

    Root cause
    Capacity exhaustion
    Component
    Database
    Contributing factor
    No alert on limitUnverified assumption
    Detection

    MediaWikiEditFailures (session loss) fired as critical at 03:03 UTC, and SessionStoreErrorRateHigh, a high 5xx rate alert, paged the batphone at 03:04. Users also reported the problem in task T390512.

    Remediation
    • Wiped the sessionstore storage, removing all sessions, and restarted Cassandra
    Action items
    • Page when the disk utilisation trend becomes unreasonable
    • Increase sessionstore storage capacity
    • Add namespacing to sessionstore
    • Add observability of the sessionstore workload
    • Identify the cause of the high rate of storage growth in sessionstore Cassandra
    • Consider an alert, without paging, on the Cassandra error rate
    • Consider paging or alerting when Cassandra nodes enter a down state
    Lessons

    The storage growth before the node failures made their timing predictable, so an alert on high storage utilisation well in advance would have given a chance to act before users were affected, and an alert on Cassandra node failures would have told responders more than an HTTP error rate alert. A cluster seen as heavily over-provisioned can still grow quickly and unsustainably under an unusual workload.

    View case
  2. IL-0232 Calico pods in the codfw WikiKube cluster hit their resource limits and crash-looped, taking nodes out one by one
    Wikimedia Foundation2024
    Symptoms

    At 05:01 UTC on 9 October 2024 a page fired for GatewayBackendErrorsHigh, elevated 5xx errors from page-analytics_cluster in codfw, followed by a growing number of KubernetesCalicoDown alerts for WikiKube nodes not running the calico-node pod and by MediaWikiMemcachedHighErrorRate. For about an hour uncached MediaWiki requests, such as those of logged-in users, edits and API calls, saw high latency or errors.

    High error rateHigh latencyContainer crash loopProbe failuresControl plane lag
    Trigger

    An operator used the new mwscript-k8s utility to run a batch of video transcode maintenance, the TimedMediaHandler requeueTranscodes.php script. mwscript-k8s deploys a whole helm release for every script invocation, so the batch created close to 2k Calico NetworkPolicies along with many other resources over several hours.

    Root cause
    Capacity exhaustion
    Component
    CalicoHelmJobs and CronJobs
    Contributing factor
    Change with a delayed effectNo runbook
    Detection

    Automated monitoring detected the issue and paged the on-call SRE: GatewayBackendErrorsHigh at 05:01:36 UTC, 16 KubernetesCalicoDown alerts at 05:03:32 UTC and MediaWikiMemcachedHighErrorRate at 05:04:15 UTC.

    Remediation
    • Raised the resource limits of the Calico and Typha pods and applied the change with helmfile, which redeployed all Calico pods
    Action items
    • Stop mwscript-k8s from creating so many resources
    Lessons

    The incident came close to a full outage. Automated monitoring caught it and many responders found the cause, but end users were affected and responders were unsure whether they could switch over to eqiad. The problem was noticed just before the hand-off from Americas to EMEA SRE shifts, a period with a long-known lack of coverage, and it helped that the transcodes were not queued on a Friday evening in the US.

    View case
Check your cluster for this

These 2 cases involve Calico and Database. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion2 cases

Component
Calico1Database1Helm1Jobs and CronJobs1
Contributing factor
Unverified assumption1Change with a delayed effect1No alert on limit1No runbook1
Organisation
Wikimedia Foundation2
  1. IL-0298 Wikimedia edits failed for 38 minutes when sessionstore Cassandra nodes in both data centres crashed on full disks
    Wikimedia Foundation2025
  2. IL-0232 Calico pods in the codfw WikiKube cluster hit their resource limits and crash-looped, taking nodes out one by one
    Wikimedia Foundation2024