Incident Library
Applied filters ComponentContainer memory limitSymptomMonitoring outageFactorSingle point of failureClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0169 The Scaleway Kubernetes API in FR-PAR ran out of memory under simultaneous deployments, and because it also handled node authentication, Kapsule and Kosmos nodes went NotReady
    Scaleway2024
    Symptoms

    At 23:48 CET on 24 January 2024 the Scaleway Kubernetes API for the FR-PAR region ran out of memory. At the same minute Scaleway saw requests to its API Gateway spike.

    Node NotReadyOOMKilledIntermittent timeoutsMonitoring outage
    Trigger

    Simultaneous deployments in the FR-PAR region pushed the Scaleway Kubernetes API out of memory.

    Root cause
    Capacity exhaustion
    Component
    Container memory limitkubelet
    Contributing factor
    Single point of failureRetry storm
    Detection

    Increased load on the API Gateway was detected at 23:48, the minute the incident began, and the incident was escalated to on-call engineers at 00:00:10.

    Remediation
    • raised the memory limit and garbage collection thresholds of the Kubernetes managed services
    • deployed additional replicas of key services
    • purged the registry cache and monitored the clusters as they stabilised
    • adjusted service configurations after the incident to keep the platform stable
    Action items
    • Implement a local authentication cache
    • Further develop node authentication
    • Enhance monitoring, adjust rate limiting strategies and improve incident response protocols
    Lessons

    Because the Kubernetes API also handled node authentication, its memory exhaustion made some nodes NotReady, and the endless retries of those nodes added load on the API Gateway. Scaleway plans a local authentication cache and further work on node authentication.

    View case
Check your cluster for this

This case involves kubelet and Container memory limit. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion1 case

Component
kubeletContainer memory limit
Contributing factor
Retry stormSingle point of failure
Organisation
Scaleway
  1. IL-0169 The Scaleway Kubernetes API in FR-PAR ran out of memory under simultaneous deployments, and because it also handled node authentication, Kapsule and Kosmos nodes went NotReady
    Scaleway2024