Cases13 cases

1 to 5 of 13

1. IL-0158 An expired credential caused authentication failures that stopped LKE and LKE-E from deploying clusters and nodes in all regions, which also blocked node recycling and cluster autoscaling

   Linode2026

   Symptoms

   From 11:45 UTC on 16 April 2026, customers of Linode Kubernetes Engine (LKE) and Linode Kubernetes Engine Enterprise (LKE-E) could not deploy new clusters or add new nodes to existing clusters. Because no new node could be deployed, node recycling and cluster autoscaling did not work either. The problem hit every region where LKE runs.

   Deploys failingScale-up failureCredentials unavailable

   Trigger

   A credential expired, which caused authentication failures when LKE and LKE-E deployed clusters and nodes. One status update describes it as an expired certificate.

   Root cause

   Certificate or credential expiry

   Component

   Node pool

   Contributing factor

   –

   Detection

   The first status page update, saying the team was investigating an emerging issue affecting LKE and LKE-E in all regions, was posted at 13:20 UTC.

   Remediation

   • created new credentials and deployed them, rotating the expired credential (certificate)

   Action items

   • Investigate what led to the credential expiry and take preventive actions

   Lessons

   One expired credential stopped every operation that needs a new node, in all regions at once: new clusters, added nodes, node recycling and autoscaling.

   View case
2. IL-0098 After 2i2c moved from ingress-nginx to nginx-ingress, a hub's Ingress took priority over cert-manager's temporary Ingress for the same host, so new hubs could not get TLS certificates

   2i2c2026

   Symptoms

   An attempt to launch a new hub failed because its ingress could not obtain a new TLS certificate. The day before, the same engineer had recreated the support deployment on the 2i2c-aws-us cluster and seen its TLS certificate orders stuck pending.

   Misrouted requestsDeploys failing

   Trigger

   Creating new hub deployments that needed new TLS certificates on clusters recently migrated from ingress-nginx to nginx-ingress.

   Root cause

   Breaking change in an upgrade

   Component

   Ingress controllercert-manager

   Contributing factor

   Change with a delayed effect

   Detection

   An attempt to launch a new hub failed; the engineer had seen certificate orders stuck pending on another cluster the day before and declared an incident.

   Remediation

   • annotated the Ingress objects that need TLS so that cert-manager could edit them in place to complete the ACME challenge

   Action items

   • Follow-up tracked in https://github.com/2i2c-org/infrastructure/issues/8016
   • Follow-up tracked in https://github.com/2i2c-org/infrastructure/issues/3278

   Lessons

   Swapping one ingress controller for another can change how Ingress objects that share a host are combined. cert-manager's temporary challenge Ingress depends on that behaviour, so the breakage shows up when a deployment needs a new certificate. 2i2c notes that it got lucky: it had already learned of this possible cause while testing on an internal cluster, and changing the cert-manager configuration was quick.

   View case
3. IL-0204 An etcd memory spike overloaded the control plane of a Render Frankfurt cluster, and DNS then failed because CoreDNS ran on the rebooted control plane nodes and could not start without the API server

   Render2024

   Symptoms

   In January 2022 an etcd memory spike overloaded the Kubernetes control plane of one of Render's clusters in Frankfurt. At first only operations that need the control plane failed: new deploys and cron job runs.

   Control plane unresponsiveDeploys failingDNS resolution failuresHigh error rate

   Trigger

   A memory spike in etcd overloaded the cluster's control plane.

   Root cause

   Capacity exhaustion

   Component

   etcdCoreDNSKubernetes API server

   Contributing factor

   Unreviewed defaultsHarmful manual action

   Action items

   • Run CoreDNS on data plane nodes instead of control plane nodes
   • Run etcd on dedicated nodes

   Lessons

   CoreDNS can keep answering from its in-memory cache while the API server is down, but it cannot initialise without it. With CoreDNS on the control plane nodes, a reboot or restart during a control plane outage turned a failure of deploys and cron jobs into a failure of requests to services that were already running. The core problem was that the data plane was not isolated enough from the control plane: even without the reboot, the etcd memory spike alone could have forced CoreDNS to restart.

   View case
4. IL-0130 Deployments of large models from Vertex Model Garden failed with "Model server never became ready" in most regions for 13 days after a storage provisioning configuration change

   Google Cloud2024

   Symptoms

   From 22 October 2024 at 14:08 US/Pacific, customers deploying large models, those that need more than 100 GB of disk, through Vertex AI Online Prediction (Vertex Model Garden deployments) had their deployments fail with the error "Model server never became ready".

   Deploys failing

   Trigger

   A recent change introduced an internal storage provisioning configuration error.

   Root cause

   Configuration change

   Component

   –

   Contributing factor

   –

   Detection

   The incident began on 22 October 2024 at 14:08 but was first posted on the status page on 4 November 2024 at 13:17 US/Pacific, about 13 days later. The first post said large Vertex Model Garden deployments (greater than 100GB) on a GKE Autopilot cluster might fail.

   Remediation

   • rolled back the configuration change that introduced the storage provisioning error

   Lessons

   A configuration error that affects only one class of deployments, here models needing more than 100 GB of disk, can stay in place across most regions for almost two weeks. The visible symptom, a model server that never becomes ready, does not point at storage provisioning, and the fix was to roll the configuration back. The report lists no follow-up actions.

   View case
5. IL-0107 Kyverno stuck in a crash loop behind a webhook with failurePolicy fail blocked every change to Kubernetes Service objects for 88 minutes

   DFDS2022

   Symptoms

   At 11:38 UTC on 3 August 2022 an end-user team told the platform team that applying a Service resource to one of the Kubernetes clusters failed.

   Deploys failingContainer crash loop

   Trigger

   A certificate chain in a third-party application expired, through no action of DFDS, and the outdated Kyverno installation got stuck in a crash loop.

   Root cause

   Certificate or credential expiry

   Component

   Policy engineAdmission webhook

   Contributing factor

   Deferred upgradesUnreviewed defaultsMonitoring blind spotIgnored early signal

   Detection

   An end-user team alerted the platform team at 11:38 UTC that applying a Service resource failed; the platform team started looking into it at 11:50 UTC.

   Remediation

   • reproduced the issue in a sandbox cluster and deployed the fix there first
   • updated Kyverno to a newer Helm chart version in the affected production cluster

   Action items

   • Implement monitoring on Kyverno
   • Create internal documentation on the Kyverno setup
   • Investigate the exact certificate issue in the third-party application
   • Consider changing the failurePolicy on ClusterPolicy resources going forward

   Lessons

   Operational issues should be added to the 'Ops Day to Day' board from the start.

   View case

By root cause7