Incident Library

IL-0237 Lucidworks · 2026-05-28kubernetesai-infra

A Kubernetes patch upgrade evicted every Ray Serve head and worker pod at once, and Lucidworks hosted LLMs went down

Lucidworks AI serves its hosted models with Ray Serve, which routes inference requests through head and worker nodes. A Ray cluster normally survives a head restart because the workers keep serving. The node pool upgrades used a surge strategy that waits only for a pod to leave the old node, not for it to run on the new one: a drain counted as successful once the pod was gone, even if the pod was still stuck in "Initializing" on its new node, and the upgrade moved straight on to the next pool. All node pools were therefore cycled in rapid succession, every head and worker pod was evicted before any replacement had finished initializing, and the whole Ray cluster went down at 16:18 UTC. Lucidworks Engineering declared a Sev1 incident at 16:54 UTC.

Recovery took longer for three reasons. A replacement head node was degraded and could not pull images, and had to be deleted by hand. The 6-11 GB LLM images transferred abnormally slowly, in 30-52 minutes instead of 2-4. And a separate operation was bringing new models online at the same time, so the Ray operator's blue-green deployment strategy required both the existing and the replacement LLM deployments to be healthy before switching traffic, which meant waiting for the slow image pulls on both. The team deleted the degraded node, waited for the image pulls on the replacement nodes, redirected traffic to the new nodes and confirmed that every hosted model answered queries. All hosted models were working at 17:53 UTC, and the incident was verified as resolved at 18:07 UTC.

Severity
sev1
Scope
Hosted LLM inference for llama-3-8b-instruct, llama-3v2-3b-instruct and phi-4-multimodal-instruct returned 500 or 429 errors; search, embedding models, the Platform UI and passthrough LLM calls were not affected
Users affected
Customers using Lucidworks AI hosted LLM inference
Region
us-southcarolina
Time to detect
–
Time to resolve
1h 35m

Symptoms

From 16:18 UTC on 28 May 2026 the Lucidworks AI hosted LLMs in the us-southcarolina region were unavailable: queries to llama-3-8b-instruct, llama-3v2-3b-instruct and phi-4-multimodal-instruct returned 500 or 429 errors. Search, embedding models, the Lucidworks Platform UI and passthrough LLM calls kept working.

The outage began during a routine Kubernetes patch upgrade that cycled the node pools, and the head and worker pods of the Ray Serve deployment were evicted before any of them had finished initializing on the new nodes. One replacement head node was degraded and could not pull container images. The LLM container images, 6-11 GB each, took 30-52 minutes to transfer instead of the usual 2-4 minutes.

While the incident was open the status page attributed it to a cloud provider capacity shortfall that kept the instances hosting the models from launching again, and said the needed capacity had been secured; the postmortem gives the upgrade's node pool cycling as the cause.

Symptom Service outagePod evictionImage pull failures

Root cause

Root cause Faulty deployment

Component Node poolModel inference server
Contributing factor No staged rolloutOverlapping failures
Trigger
A routine Kubernetes patch upgrade on a cluster in us-southcarolina cycled its node pools one after another in quick succession.

Detection

–

Remediation

Action items

Lessons

–

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Node pool, Model inference server among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.