Incident Library
Applied filters Root causeFaulty deploymentSymptomService outageFactorOverlapping failuresClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0237 A Kubernetes patch upgrade evicted every Ray Serve head and worker pod at once, and Lucidworks hosted LLMs went down
    Lucidworks2026sev1
    Symptoms

    From 16:18 UTC on 28 May 2026 the Lucidworks AI hosted LLMs in the us-southcarolina region were unavailable: queries to llama-3-8b-instruct, llama-3v2-3b-instruct and phi-4-multimodal-instruct returned 500 or 429 errors. Search, embedding models, the Lucidworks Platform UI and passthrough LLM calls kept working.

    Service outagePod evictionImage pull failures
    Trigger

    A routine Kubernetes patch upgrade on a cluster in us-southcarolina cycled its node pools one after another in quick succession.

    Root cause
    Faulty deployment
    Component
    Node poolModel inference server
    Contributing factor
    No staged rolloutOverlapping failures
    Remediation
    • Deleted the degraded replacement head node that could not pull container images
    • Waited for the LLM image pulls to finish on the replacement nodes
    • Launched new nodes with the required LLM images and redirected Lucidworks AI traffic to them
    Action items
    • Upgrade clusters that host LLM workloads in sequence, with each node pool fully healthy before the next one is upgraded
    • Investigate image pre-loading, such as pre-baked disks or image streaming, for large ML model images
    • Open a support ticket with the cloud provider about the abnormal Docker pull times
    • Coordinate Kubernetes maintenance windows with LLM service owners so they do not clash with ongoing deployments
    • Build tooling to force a Ray cluster failover to the blue or green state instead of waiting for Ray to settle it
    View case
  2. IL-0278 A weekly update put a Redis Cloud management process into an infinite loop, and most nodes of a GCP cluster failed
    Redis2016
    Symptoms

    On 8 February 2016 at 17:45 UTC, four nodes of the Redis Cloud cluster us-central1-1-1.gce.garantiadata.com on Google Cloud Platform stopped responding, four minutes after a first node had failed. The databases whose endpoints end in us-central1-1-1.gce.garantiadata.com went down.

    Service outageFailover failure
    Trigger

    The weekly upgrade of Redis Cloud's non-critical cluster components reached its first Google Cloud Platform cluster, us-central1-1-1.gce.garantiadata.com, at 17:30 UTC.

    Root cause
    Faulty deployment
    Component
    In-memory cacheDatabase failover managerAuthoritative DNS servers and records
    Contributing factor
    Overlapping failures
    Remediation
    • Shut down and terminated all nodes of the cluster
    • Deleted all of the cluster's endpoints from its DNS zone, removing by hand the entries left by throttled deletes
    • Provisioned new cluster nodes from scratch without the weekly update
    • Reconnected the original nodes' storage to the new nodes so persistence-enabled databases could be recovered
    • Bootstrapped and launched the databases, recovering the ones with persistence from storage
    Action items
    • Suspend all automatic weekly upgrades until a procedure that prevents this kind of outage is in place
    • Revalidate the QA processes for DevOps code deployments
    • Improve the tooling so it handles throttling of DNS requests
    View case
Check your cluster for this

These 2 cases involve In-memory cache and Authoritative DNS servers and records. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Faulty deployment2 cases

Component
In-memory cache1Authoritative DNS servers and records1Database failover manager1Model inference server1Node pool1
Contributing factor
Overlapping failures2No staged rollout1
Organisation
Lucidworks1Redis1
  1. IL-0237 A Kubernetes patch upgrade evicted every Ray Serve head and worker pod at once, and Lucidworks hosted LLMs went down
    Lucidworks2026
  2. IL-0278 A weekly update put a Redis Cloud management process into an infinite loop, and most nodes of a GCP cluster failed
    Redis2016