Cases2 cases

1 to 2 of 2

1. IL-0169 The Scaleway Kubernetes API in FR-PAR ran out of memory under simultaneous deployments, and because it also handled node authentication, Kapsule and Kosmos nodes went NotReady

   Scaleway2024

   Symptoms

   At 23:48 CET on 24 January 2024 the Scaleway Kubernetes API for the FR-PAR region ran out of memory. At the same minute Scaleway saw requests to its API Gateway spike.

   Node NotReadyOOMKilledIntermittent timeoutsMonitoring outage

   Trigger

   Simultaneous deployments in the FR-PAR region pushed the Scaleway Kubernetes API out of memory.

   Root cause

   Capacity exhaustion

   Component

   Container memory limitkubelet

   Contributing factor

   Single point of failureRetry storm

   Detection

   Increased load on the API Gateway was detected at 23:48, the minute the incident began, and the incident was escalated to on-call engineers at 00:00:10.

   Remediation

   • raised the memory limit and garbage collection thresholds of the Kubernetes managed services
   • deployed additional replicas of key services
   • purged the registry cache and monitored the clusters as they stabilised
   • adjusted service configurations after the incident to keep the platform stable

   Action items

   • Implement a local authentication cache
   • Further develop node authentication
   • Enhance monitoring, adjust rate limiting strategies and improve incident response protocols

   Lessons

   Because the Kubernetes API also handled node authentication, its memory exhaustion made some nodes NotReady, and the endless retries of those nodes added load on the API Gateway. Scaleway plans a local authentication cache and further work on node authentication.

   View case
2. IL-0040 Parallel Helm installs saturated small gp2 root volumes on a kops cluster and left all three workers NotReady with the kubelet or Docker down

   Saltside2017

   Symptoms

   While running the CI process on a local machine to test an application Helm chart, the SRE lead watched the pods with watch kubectl get pods. All pods reached Running, and then some went back to Pending. Two hours later the problem had grown and some pods showed the phase Unknown.

   Pods PendingImage pull failuresNode NotReadyService outage

   Trigger

   A CI run installed the application's Helm charts for all four markets (Efritin.com, Tonaton.com, Ikman.lk and Bikroy.com) on the nonproduction cluster at the same time.

   Root cause

   Capacity exhaustion

   Component

   Node root diskkubeletContainer runtimeHelm

   Contributing factor

   Unreviewed defaultsMonitoring blind spotIgnored early signalSingle point of failure

   Detection

   The SRE lead saw pods go from Running back to Pending while watching kubectl get pods during the CI run; no monitoring raised an alert, even when all three workers were not ready.

   Remediation

   • terminated the three worker nodes and let the auto scaling group replace them

   Action items

   • Fix the Datadog integration for the nonproduction AWS account and repeat the installs with telemetry
   • Increase the worker root EBS volumes from 20GB to 120GB
   • Keep common base images pulled on every node with a DaemonSet until kops supports user hooks
   • Configure different allocation limits for the different layers in Kubernetes
   • Discuss a highly available Tiller with the Helm team
   • Find a way to throttle the number of newly created pods

   Lessons

   An ordinary deploy of the application could collapse the cluster, because the default 20GB gp2 root volumes could not keep up with parallel image pulls and container starts. Without node metrics and pod phase metrics, three NotReady workers raised no alert. A single Tiller replica can block the deploy that would fix an outage.

   View case

By root cause1