Cases20 cases

1 to 5 of 20

1. IL-0237 A Kubernetes patch upgrade evicted every Ray Serve head and worker pod at once, and Lucidworks hosted LLMs went down

   Lucidworks2026sev1

   Symptoms

   From 16:18 UTC on 28 May 2026 the Lucidworks AI hosted LLMs in the us-southcarolina region were unavailable: queries to llama-3-8b-instruct, llama-3v2-3b-instruct and phi-4-multimodal-instruct returned 500 or 429 errors. Search, embedding models, the Lucidworks Platform UI and passthrough LLM calls kept working.

   Service outagePod evictionImage pull failures

   Trigger

   A routine Kubernetes patch upgrade on a cluster in us-southcarolina cycled its node pools one after another in quick succession.

   Root cause

   Faulty deployment

   Component

   Node poolModel inference server

   Contributing factor

   No staged rolloutOverlapping failures

   Remediation

   • Deleted the degraded replacement head node that could not pull container images
   • Waited for the LLM image pulls to finish on the replacement nodes
   • Launched new nodes with the required LLM images and redirected Lucidworks AI traffic to them

   Action items

   • Upgrade clusters that host LLM workloads in sequence, with each node pool fully healthy before the next one is upgraded
   • Investigate image pre-loading, such as pre-baked disks or image streaming, for large ML model images
   • Open a support ticket with the cloud provider about the abnormal Docker pull times
   • Coordinate Kubernetes maintenance windows with LLM service owners so they do not clash with ongoing deployments
   • Build tooling to force a Ray cluster failover to the blue or green state instead of waiting for Ray to settle it

   View case
2. IL-0166 A compute host update removed the credential parser used to decode StackScripts, so new Linodes and every new LKE and LKE-E node failed to boot and autoscaling and node recycling failed in all regions

   Linode2026

   Symptoms

   Linodes deployed with StackScripts failed to deploy and could not boot. Every newly created Linode Kubernetes Engine (LKE) and LKE-E node relies on StackScripts, so LKE autoscaling and node recycling failed as well, in all regions.

   Scale-up failure

   Trigger

   A recent update on the compute hosts removed a credential parser that is used when decoding StackScripts.

   Root cause

   Faulty deployment

   Component

   Cluster autoscalerNode pool

   Contributing factor

   –

   Detection

   The issue was identified at 18:18 UTC on 2026-05-15. The first status post went up at 00:42 UTC on 2026-05-16.

   Remediation

   • tested and applied a rollback starting at 01:10 UTC on 2026-05-16, and failures dropped from 01:20 UTC

   Lessons

   Removing a credential parser from the compute hosts broke the boot of every Linode deployed with StackScripts, and every new LKE and LKE-E node is one of them, so node creation, autoscaling and node recycling stopped in all regions. The rollback reduced failures ten minutes after testing began.

   View case
3. IL-0234 An onboarding operator deployed without its scope file installed dev applications on other IONOS Kubernetes clusters

   IONOS Cloud2026

   Symptoms

   On 8 May 2026 IONOS Cloud reported "K8s Control Planes not available": some Managed Kubernetes customers in DE/TXL, DE/FRA and DE/FRA/2 had trouble connecting to the control plane and saw degraded Kubernetes functionality. Between 15:22 and 16:12 (UTC+2) applications meant for a new internal development cluster were installed on other customer clusters, which were also modified.

   Control plane unresponsiveImage pull failures

   Trigger

   A routine internal operation onboarded a new Managed Kubernetes development cluster. The scoping configuration file for the onboarding operator failed to reach the deployment branch because of a merge conflict, and at the same time a second change removed the operator from the exclusion list, which deployed it at once.

   Root cause

   Faulty deployment

   Component

   CI systemContainer registry and its credentials

   Contributing factor

   No configuration testsExcessive blast radiusMissed in review

   Remediation

   • Corrected the onboarding operator so it needs an explicit namespace configuration before acting on any cluster
   • Reconfigured the deployment system to fail deployments when required configuration files are missing
   • Rolled back the identified changes on the affected resources
   • Ran a recovery script, validated in staging, that removed all unintended installations and updates from the affected clusters

   Action items

   • Add a mandatory CI/CD step that confirms all required configuration files reached the target branch before any deployment starts
   • Require additional peer review and change management approval for every deployment that touches production, whatever its size
   • Give teams a way to inform each affected customer in time about the status of post-incident cleanup

   Lessons

   IONOS treats the unintended change to cluster configurations as a failure of the controls that should keep internal operations from affecting customer environments.

   View case
4. IL-0167 A phased rollout of an LKE software release in Osaka stalled LKE-E cluster creation, version upgrades and control plane ACL changes

   Linode2026

   Symptoms

   From about 19:15 UTC on 12 March 2026, customers in the Osaka data center (JP-OSA) could not create Linode Kubernetes Engine for Enterprise (LKE-E) clusters. Cluster creation stalled indefinitely.

   Cluster or node provisioning stallUpgrade stall

   Trigger

   A phased rollout of the LKE software release reached the Osaka data center.

   Root cause

   Faulty deployment

   Component

   Managed Kubernetes control plane and provider API

   Contributing factor

   –

   Detection

   The problem began at about 19:15 UTC, and the first status post, about five and a half hours later at 00:45 UTC on 13 March, said the team was investigating an emerging issue with deploying new LKE Enterprise clusters in Osaka.

   Remediation

   • deployed a software fix to mitigate the impact

   Action items

   • Investigate the root cause further and take preventive actions

   Lessons

   The problem followed a phased rollout of an LKE software release in the Osaka data center. There, cluster creation and administrative tasks stalled for more than six hours, and the first status post came about five and a half hours after the start.

   View case
5. IL-0160 A repaired legacy GPU monitoring script sent a firmware query that triggers a race condition in the GPU System Processor of NVIDIA RTX 4000 Ada GPUs, locking up GPU Linodes and LKE worker nodes in several regions

   Linode2026

   Symptoms

   From about 21:00 UTC on March 4, 2026, customers using Linodes backed by NVIDIA RTX 4000 Ada GPUs saw the GPUs lock up. At first the problem seemed limited to worker nodes of the Linode Kubernetes Engine (LKE); later Linode confirmed that every Linode with this hardware was affected, while Standard Compute and other GPU instances were not.

   GPU unavailableControl plane unresponsive

   Trigger

   Deployment of a repaired legacy GPU monitoring script, fixed during a routine improvement effort to restore a missing metric on internal observability dashboards.

   Root cause

   Faulty deployment

   Component

   Telemetry agentGPU driver

   Contributing factor

   Missed in reviewUnverified assumption

   Detection

   Customers saw the lockups from about 21:00 UTC; the first status post went up at 02:07 UTC on March 5, about 5 hours after the lockups began.

   Remediation

   • Disabled the monitoring script across the GPU fleet
   • Rebooted the affected nodes

   Action items

   • Take improvement and prevention measures to stop a recurrence; the postmortem does not list them

   Lessons

   A monitoring script can take down the hardware it watches. A fix meant only to restore a missing metric brought back a firmware query written for an earlier GPU generation, and on the RTX 4000 Ada architecture that query against an active GPU triggers a lockup. The script ran across the GPU fleet, so every Linode with this GPU was affected in several regions, and the investigation first suspected a regression in the host hypervisor or the GPU firmware.

   View case

By root cause1