Cases6 cases

3 to 6 of 6

1. IL-0168 Simultaneous cluster auto-upgrades overloaded Scaleway's Kubernetes API, and a node authentication retry loop kept Kapsule and Kosmos nodes NotReady in FR-PAR

   Scaleway2024

   Symptoms

   At 01:04 CET on 13 February 2024, the Kubernetes API behind Scaleway's managed Kubernetes services Kapsule and Kosmos in the Paris region (FR-PAR) spiked in RAM and CPU use and ran out of memory. Nodes in FR-PAR could no longer authenticate to their control plane, and some of them went "Not Ready". Customers on both shared (mutualized) and dedicated environments were affected.

   Node NotReadyControl plane lagOOMKilled

   Trigger

   A new Kubernetes release was made available to all users on 12 February 2024, and during the night of 12 to 13 February many clusters triggered their auto-upgrade at the same time.

   Root cause

   Capacity exhaustion

   Component

   Kubernetes API server

   Contributing factor

   Retry stormAutomation amplificationNo staged rollout

   Detection

   The timeline's first entry, at 01:04 CET, records the Kubernetes API's RAM and CPU spikes and out-of-memory state, with nodes unable to authenticate to their control plane and some becoming "Not Ready". The first recorded action, 42 minutes later at 01:46, was to increase the resources allocated to the API.

   Remediation

   • increased the resources allocated to the Kubernetes API at 01:46 and again at 02:20
   • applied API traffic restrictions at 02:30 to smooth the incoming traffic
   • removed the traffic restrictions progressively in 50-100 batches from 03:20
   • ran a script at 03:40 to replace the failing nodes created during the congestion
   • rolled back to the former node authentication mechanism, first on a subset of clusters from 15:45 and on all clusters by 19:41 on 19 February
   • disabled cluster upgrade and configuration update features until 10:30 on 20 February

   Action items

   • Increase the resources allocated to the Kubernetes API
   • Roll back to the former node authentication mechanism
   • Implement a local authentication cache on nodes
   • Rework the resource allocation strategy to behave better under load spikes
   • Add new request queuing mechanisms for Kubernetes
   • Separate the different components to reduce side effects under high load or failure
   • Monitor selected metrics closely to identify anomalous patterns earlier
   • Refund customers who deployed nodes during the incident but could not use them normally

   Lessons

   Letting many clusters auto-upgrade at the same moment after a release turns routine upgrades into one coordinated load spike on the shared Kubernetes API. A node authentication mechanism that retries without letting up turns a slow API into an unavailable one, and auto-healing that replaces NotReady nodes amplifies the problem. The congestion pattern had already appeared three weeks earlier, and short-term mitigations did not stop it from recurring, so Scaleway rolled back the new authentication mechanism. Scaleway notes that as the number of customer clusters and nodes grows sharply, traffic control mechanisms have to improve to prevent congestion.

   View case
2. IL-0169 The Scaleway Kubernetes API in FR-PAR ran out of memory under simultaneous deployments, and because it also handled node authentication, Kapsule and Kosmos nodes went NotReady

   Scaleway2024

   Symptoms

   At 23:48 CET on 24 January 2024 the Scaleway Kubernetes API for the FR-PAR region ran out of memory. At the same minute Scaleway saw requests to its API Gateway spike.

   Node NotReadyOOMKilledIntermittent timeoutsMonitoring outage

   Trigger

   Simultaneous deployments in the FR-PAR region pushed the Scaleway Kubernetes API out of memory.

   Root cause

   Capacity exhaustion

   Component

   Container memory limitkubelet

   Contributing factor

   Single point of failureRetry storm

   Detection

   Increased load on the API Gateway was detected at 23:48, the minute the incident began, and the incident was escalated to on-call engineers at 00:00:10.

   Remediation

   • raised the memory limit and garbage collection thresholds of the Kubernetes managed services
   • deployed additional replicas of key services
   • purged the registry cache and monitored the clusters as they stabilised
   • adjusted service configurations after the incident to keep the platform stable

   Action items

   • Implement a local authentication cache
   • Further develop node authentication
   • Enhance monitoring, adjust rate limiting strategies and improve incident response protocols

   Lessons

   Because the Kubernetes API also handled node authentication, its memory exhaustion made some nodes NotReady, and the endless retries of those nodes added load on the API Gateway. Scaleway plans a local authentication cache and further work on node authentication.

   View case
3. IL-0004 A new default pod priority let a new Cortex cluster preempt every production ingester in a cascade, taking Grafana Cloud Hosted Prometheus down

   Grafana Labs2019

   Symptoms

   On Friday 19 July 2019 the Grafana Cloud Hosted Prometheus service, which is built on the Cortex project, was down for 26 minutes. New error-budget-based Prometheus alerts paged the on-call engineer within 4 minutes of the start.

   Pod evictionHigh error rateService outageOOMKilled

   Trigger

   An engineer created a new dedicated Cortex cluster for a large customer. Its configuration did not include the new pod priorities, so all its pods got the default medium priority, and the Kubernetes cluster did not have enough free resources to fit it.

   Root cause

   Configuration change

   Component

   Pod priority and preemption

   Contributing factor

   Partial rolloutAutomation amplificationMissed in reviewNo headroomRetry storm

   Detection

   New error-budget-based Prometheus alerts paged the on-call engineer within 4 minutes of the start of the outage.

   Remediation

   • Scaled up the underlying Kubernetes cluster to fit both the new and the production Cortex clusters
   • Diagnosed and fixed the out-of-memory errors in the authenticating reverse proxies in front of Cortex

   Action items

   • Give the production ingesters the high priority class, which should have been done before medium became the default
   • Have more people review changes to configuration that is global to the cluster
   • Require a design document for every configuration change outside a namespace
   • Automate the sizing of the frontend authenticating reverse proxy so it does not run out of memory under overload
   • Investigate the default backoff and scaling behaviour of Prometheus to prevent a stampeding herd of retries

   Lessons

   Medium should not have become the default priority until the production ingesters had the high priority, and that designation should have reached production sooner. The change was considered too minor for a design document and was only discussed in a GitHub issue; cluster-global configuration needs more scrutiny. On the positive side, once Cortex had enough capacity it recovered by itself, and Grafana Loki helped confirm that the ingesters exited gracefully.

   View case
4. IL-0001 Cluster-wide DNS outage after CoreDNS pods were OOM-killed again and again under a burst of DNS queries

   Zalando2019

   Symptoms

   Customers of the Zalando Fashion Store got a high rate of errors on all web product pages and outfit pages for more than an hour. The errors came from the main data aggregation layer service, which runs in one of Zalando's Kubernetes clusters.

   High error rateDNS resolution failuresOOMKilledMonitoring outage

   Trigger

   Calls from the aggregation layer to a single, not very important downstream service started timing out. The aggregation layer returned 404s for them, clients retried, and the resulting spike in requests became a spike in DNS queries to CoreDNS.

   Root cause

   Capacity exhaustion

   Component

   CoreDNSContainer memory limitClient-side DNS resolver and cache

   Contributing factor

   Unknown limitNo cachingRetry stormUnreviewed defaultsExcessive blast radiusMonitoring blind spot

   Detection

   The on-call engineer for the aggregation layer phoned the on-call Kubernetes engineer. Automatic paging did not happen because the cluster's monitoring and alerting ran inside the cluster and failed along with DNS. The Kubernetes engineer then saw that the CoreDNS pods were being OOM-killed.

   Remediation

   • Raised the CoreDNS memory requests and limits by hand from 100Mi to 2000Mi

   Action items

   • Introduce a more resilient DNS setup: CoreDNS fronted by dnsmasq on every node
   • Improve high-level alerting from outside the clusters, so that a cluster problem is known before or at the same time as applications are affected
   • Improve incident response processes and tooling so that any on-call engineer can escalate directly to other teams when automatic paging fails, for example through a chat bot

   Lessons

   The Kubernetes DNS setup was far from resilient enough: a single application could take down DNS for everything in the cluster. The blast radius of one application sending more DNS queries should be limited to one node at most, ideally to single pods. Known Kubernetes DNS problems, such as Linux kernel races in conntrack handling and the ndots 5 problem, were familiar to the team and partly mitigated, but their combined effect on the DNS infrastructure was worse than the team had understood. Monitoring that runs inside the cluster was not robust enough: when the cluster's DNS broke, it could not page anyone, and recovery took longer than it should have.

   View case

By root cause2