Cases21 cases

6 to 10 of 21

1. IL-0201 A framework upgrade caused kernel panics in GitHub's Kubernetes infrastructure, and GitHub.com web and API requests, mostly for issues and pull requests, failed intermittently for about 16 hours

   GitHub2025

   Symptoms

   From 18:05 UTC on March 17, 2025, GitHub.com web and API requests failed now and then for a small percentage of users, mostly requests about issues and pull requests. At 18:39 UTC the status page reported degraded performance for Issues.

   High error rateIntermittent timeouts

   Trigger

   A framework upgrade rolled out to GitHub's Kubernetes infrastructure caused kernel panics.

   Root cause

   Breaking change in an upgrade

   Component

   Linux kernel

   Contributing factor

   –

   Detection

   The status page opened the incident at 18:39 UTC, 34 minutes after the failures began, as an investigation of reports of degraded performance for Issues.

   Remediation

   • Downgraded the framework upgrade that caused the kernel panics
   • Disabled the problematic feature of the framework

   Action items

   • Investigate why the upgrade caused the kernel panics
   • Take steps to temporarily prevent the problem from coming back
   • Work out longer-term patch plans
   • Improve observability so that similar classes of problems can be reacted to quickly

   View case
2. IL-0081 API requests timed out during deployment rollouts because the ingress-nginx enable-serial-reloads option held back endpoint updates while a long NGINX reload was running

   Cresta2025

   Symptoms

   During a deployment, traffic into the Kubernetes cluster started hitting dead ends: some API requests timed out and requests were slow and disrupted while the rollout ran.

   Intermittent timeoutsStale endpoint routingHigh latency

   Trigger

   A deployment rollout changed backend endpoints while a long-running NGINX full reload was in progress, with the enable-serial-reloads option enabled on the ingress-nginx controller.

   Root cause

   Software defect

   Component

   Ingress controller

   Contributing factor

   –

   Remediation

   • restarted the NGINX Ingress controller as an attempt to resolve the problem

   Action items

   • Open an issue in kubernetes/ingress-nginx describing how enable-serial-reloads blocks dynamic endpoint updates

   Lessons

   A flag meant only to serialise full reloads also delayed endpoint updates, which opened a window during rollouts in which the ingress routed to pods that no longer existed. Options that change how the ingress controller applies configuration need to be checked against what happens to endpoint updates during a long reload.

   View case
3. IL-0131 A new kind of GenerateContent request made Vertex AI prediction API servers segfault, and once enough of them restarted at the same time the serving fleet could not carry the load

   Google Cloud2024

   Symptoms

   From 12:06 US/Pacific on 12 June 2024, services built on Vertex AI Prediction began reporting failed user requests in eight regions: us-central1, asia-southeast1, europe-west3, europe-west4, us-east1, us-west1, northamerica-northeast1 and us-east4.

   High error rateHigh latencyIntermittent timeouts

   Trigger

   Starting on 8 June 2024, a new form of user request to the GenerateContent API caused intermittent segmentation faults in the Vertex Serving Prediction API servers; on 12 June the rate of such requests increased.

   Root cause

   Software defect

   Component

   Model inference serverCloud load balancer

   Contributing factor

   Partial rolloutNo headroom

   Detection

   Internal production monitoring alerted engineers to the segfaults on 10 June 2024 at 07:35 US/Pacific, before any visible customer impact; on 12 June services built on Vertex Prediction reported failing user requests.

   Remediation

   • confirmed the new failures came from the same segfault found on 10 June
   • accelerated the rollout of the fix that was already in progress, completed at 15:59 US/Pacific

   Action items

   • Ensure early signals of server binary issues are captured in production health analysis and investigated in time
   • Ensure release validation of feature changes and updates in production

   Lessons

   A crash that only some requests trigger can stay invisible while load balancers route around the restarting servers. It becomes an outage when the share of such requests grows until too many servers are restarting at once. Here the cause was known and a fix was rolling out two days before customers were affected, but the progressive rollout had not finished when the triggering traffic increased; Google's follow-up is to investigate early signals of server binary issues in time.

   View case
4. IL-0161 A faulty earlier API update made new LKE cluster provisioning in Toronto time out for three days, leaving clusters with no node pools or the wrong number of nodes

   Linode2024

   Symptoms

   At about 14:05 UTC on 22 April 2024, Linode's team noticed that new LKE clusters were failing to provision in the Toronto data center.

   Cluster or node provisioning stallIntermittent timeouts

   Trigger

   A previous update to the API did not work as intended and caused timeouts when new LKE clusters were provisioned.

   Root cause

   Software defect

   Component

   Node pool

   Contributing factor

   –

   Detection

   Linode's team noticed new LKE clusters failing to provision in the Toronto data center at about 14:05 UTC on 22 April 2024.

   Remediation

   • disabled the affected section of Toronto infrastructure from provisioning new LKE clusters
   • implemented a fix for the faulty API update
   • put the disabled infrastructure back into use after the fix

   Lessons

   The fault was in an earlier API change, not in the infrastructure that showed it, but most failures concentrated on one backend resource in one data center. Taking that resource out of provisioning limited the impact while the root cause was found, which took three days.

   View case
5. IL-0169 The Scaleway Kubernetes API in FR-PAR ran out of memory under simultaneous deployments, and because it also handled node authentication, Kapsule and Kosmos nodes went NotReady

   Scaleway2024

   Symptoms

   At 23:48 CET on 24 January 2024 the Scaleway Kubernetes API for the FR-PAR region ran out of memory. At the same minute Scaleway saw requests to its API Gateway spike.

   Node NotReadyOOMKilledIntermittent timeoutsMonitoring outage

   Trigger

   Simultaneous deployments in the FR-PAR region pushed the Scaleway Kubernetes API out of memory.

   Root cause

   Capacity exhaustion

   Component

   Container memory limitkubelet

   Contributing factor

   Single point of failureRetry storm

   Detection

   Increased load on the API Gateway was detected at 23:48, the minute the incident began, and the incident was escalated to on-call engineers at 00:00:10.

   Remediation

   • raised the memory limit and garbage collection thresholds of the Kubernetes managed services
   • deployed additional replicas of key services
   • purged the registry cache and monitored the clusters as they stabilised
   • adjusted service configurations after the incident to keep the platform stable

   Action items

   • Implement a local authentication cache
   • Further develop node authentication
   • Enhance monitoring, adjust rate limiting strategies and improve incident response protocols

   Lessons

   Because the Kubernetes API also handled node authentication, its memory exhaustion made some nodes NotReady, and the endless retries of those nodes added load on the API Gateway. Scaleway plans a local authentication cache and further work on node authentication.

   View case

By root cause7