Incident Library
Applied filters DomainKubernetesOrganisationLucidworksClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0237 A Kubernetes patch upgrade evicted every Ray Serve head and worker pod at once, and Lucidworks hosted LLMs went down
    Lucidworks2026sev1
    Symptoms

    From 16:18 UTC on 28 May 2026 the Lucidworks AI hosted LLMs in the us-southcarolina region were unavailable: queries to llama-3-8b-instruct, llama-3v2-3b-instruct and phi-4-multimodal-instruct returned 500 or 429 errors. Search, embedding models, the Lucidworks Platform UI and passthrough LLM calls kept working.

    Service outagePod evictionImage pull failures
    Trigger

    A routine Kubernetes patch upgrade on a cluster in us-southcarolina cycled its node pools one after another in quick succession.

    Root cause
    Faulty deployment
    Component
    Node poolModel inference server
    Contributing factor
    No staged rolloutOverlapping failures
    Remediation
    • Deleted the degraded replacement head node that could not pull container images
    • Waited for the LLM image pulls to finish on the replacement nodes
    • Launched new nodes with the required LLM images and redirected Lucidworks AI traffic to them
    Action items
    • Upgrade clusters that host LLM workloads in sequence, with each node pool fully healthy before the next one is upgraded
    • Investigate image pre-loading, such as pre-baked disks or image streaming, for large ML model images
    • Open a support ticket with the cloud provider about the abnormal Docker pull times
    • Coordinate Kubernetes maintenance windows with LLM service owners so they do not clash with ongoing deployments
    • Build tooling to force a Ray cluster failover to the blue or green state instead of waiting for Ray to settle it
    View case
Check your cluster for this

This case involves Model inference server and Node pool. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Faulty deployment1 case

Component
Model inference serverNode pool
Contributing factor
No staged rolloutOverlapping failures
Organisation
Lucidworks
  1. IL-0237 A Kubernetes patch upgrade evicted every Ray Serve head and worker pod at once, and Lucidworks hosted LLMs went down
    Lucidworks2026