Incident Library

Organisation

Scaleway

Scaleway · scaleway

2 cases · 2024

Root cause Capacity exhaustion2
Component Kubernetes API server1kubelet1Container memory limit1
Contributing factor Retry storm2Automation amplification1No staged rollout1Single point of failure1
Symptom Node NotReady2OOMKilled2Control plane lag1Intermittent timeouts1Monitoring outage1

Cases

  1. IL-0168 Simultaneous cluster auto-upgrades overloaded Scaleway's Kubernetes API, and a node authentication retry loop kept Kapsule and Kosmos nodes NotReady in FR-PARScaleway · 2024

    Root cause Capacity exhaustion

  2. IL-0169 The Scaleway Kubernetes API in FR-PAR ran out of memory under simultaneous deployments, and because it also handled node authentication, Kapsule and Kosmos nodes went NotReadyScaleway · 2024

    Root cause Capacity exhaustion