Incident Library
Applied filters SymptomContainer crash loopFactorChange with a delayed effectDomainKubernetesClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0232 Calico pods in the codfw WikiKube cluster hit their resource limits and crash-looped, taking nodes out one by one
    Wikimedia Foundation2024
    Symptoms

    At 05:01 UTC on 9 October 2024 a page fired for GatewayBackendErrorsHigh, elevated 5xx errors from page-analytics_cluster in codfw, followed by a growing number of KubernetesCalicoDown alerts for WikiKube nodes not running the calico-node pod and by MediaWikiMemcachedHighErrorRate. For about an hour uncached MediaWiki requests, such as those of logged-in users, edits and API calls, saw high latency or errors.

    High error rateHigh latencyContainer crash loopProbe failuresControl plane lag
    Trigger

    An operator used the new mwscript-k8s utility to run a batch of video transcode maintenance, the TimedMediaHandler requeueTranscodes.php script. mwscript-k8s deploys a whole helm release for every script invocation, so the batch created close to 2k Calico NetworkPolicies along with many other resources over several hours.

    Root cause
    Capacity exhaustion
    Component
    CalicoHelmJobs and CronJobs
    Contributing factor
    Change with a delayed effectNo runbook
    Detection

    Automated monitoring detected the issue and paged the on-call SRE: GatewayBackendErrorsHigh at 05:01:36 UTC, 16 KubernetesCalicoDown alerts at 05:03:32 UTC and MediaWikiMemcachedHighErrorRate at 05:04:15 UTC.

    Remediation
    • Raised the resource limits of the Calico and Typha pods and applied the change with helmfile, which redeployed all Calico pods
    Action items
    • Stop mwscript-k8s from creating so many resources
    Lessons

    The incident came close to a full outage. Automated monitoring caught it and many responders found the cause, but end users were affected and responders were unsure whether they could switch over to eqiad. The problem was noticed just before the hand-off from Americas to EMEA SRE shifts, a period with a long-known lack of coverage, and it helped that the transcodes were not queued on a Friday evening in the US.

    View case
  2. IL-0467 Boskos crash-looped after a node drain started it from a buggy StatefulSet image, failing Kubernetes PR jobs for 9 hours
    Kubernetes project2019
    Symptoms

    Overnight into 26 March 2019 (PDT), every Prow test job that needed a GCP project from Boskos, the Kubernetes test-infra service that hands out GCP projects to test jobs, failed. The required job pull-kubernetes-e2e-gce failed on every run, so no pull request could be merged in kubernetes/kubernetes for just over nine hours, and other projects saw similar failures.

    Container crash loopService outage
    Trigger

    At 23:04 PDT the node running the only Boskos pod was drained to be replaced with a 1.13 node during the upgrade of the Prow clusters from Kubernetes 1.11 to 1.13. The replacement pod started from the StatefulSet's current spec, which named a newer Boskos image containing the bug.

    Root cause
    Faulty deployment
    Component
    CI systemNode pool
    Contributing factor
    Change with a delayed effectUnverified assumptionSingle point of failureNo alert on limitChange started before off-hours
    Detection

    Users reported the failure on GitHub at 02:50 and on Slack at 05:16 and 06:27, and an engineer saw the Slack reports on waking at 07:16. The Boskos monitoring dashboard showed the failure, but no alert caught it.

    Remediation
    • Built and deployed a new Boskos image with the bug fixed
    • Cleaned up the GCP projects that had been left open
    Action items
    • Change the Boskos configuration so that deployments actually take effect (#11956)
    • Automate Boskos deploys (#11957)
    • Improve alerting on Boskos status (#11953)
    Lessons

    The write-up lists what went poorly: a ten-hour upgrade was started shortly after 17:00 when on-call coverage ended at 18:00; deployments were not checked to confirm that something had actually rolled out; the Boskos StatefulSet was not set up to perform updates, so changing its spec left the old pod running; the December change appears never to have been tested; and Boskos was deployed by hand, unlike most of Prow, which left more room for error. Once people were available, the problem was fixed quickly.

    View case
Check your cluster for this

These 2 cases involve Calico and CI system. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause2

Capacity exhaustion1 case

Component
CalicoHelmJobs and CronJobs
Contributing factor
Change with a delayed effectNo runbook
Organisation
Wikimedia Foundation
  1. IL-0232 Calico pods in the codfw WikiKube cluster hit their resource limits and crash-looped, taking nodes out one by one
    Wikimedia Foundation2024

Faulty deployment1 case

Component
CI systemNode pool
Contributing factor
Unverified assumptionChange started before off-hoursChange with a delayed effectNo alert on limitSingle point of failure
Organisation
Kubernetes project
  1. IL-0467 Boskos crash-looped after a node drain started it from a buggy StatefulSet image, failing Kubernetes PR jobs for 9 hours
    Kubernetes project2019