Incident Library
Applied filters Root causeFaulty deploymentSymptomImage pull failuresDomainKubernetesClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0237 A Kubernetes patch upgrade evicted every Ray Serve head and worker pod at once, and Lucidworks hosted LLMs went down
    Lucidworks2026sev1
    Symptoms

    From 16:18 UTC on 28 May 2026 the Lucidworks AI hosted LLMs in the us-southcarolina region were unavailable: queries to llama-3-8b-instruct, llama-3v2-3b-instruct and phi-4-multimodal-instruct returned 500 or 429 errors. Search, embedding models, the Lucidworks Platform UI and passthrough LLM calls kept working.

    Service outagePod evictionImage pull failures
    Trigger

    A routine Kubernetes patch upgrade on a cluster in us-southcarolina cycled its node pools one after another in quick succession.

    Root cause
    Faulty deployment
    Component
    Node poolModel inference server
    Contributing factor
    No staged rolloutOverlapping failures
    Remediation
    • Deleted the degraded replacement head node that could not pull container images
    • Waited for the LLM image pulls to finish on the replacement nodes
    • Launched new nodes with the required LLM images and redirected Lucidworks AI traffic to them
    Action items
    • Upgrade clusters that host LLM workloads in sequence, with each node pool fully healthy before the next one is upgraded
    • Investigate image pre-loading, such as pre-baked disks or image streaming, for large ML model images
    • Open a support ticket with the cloud provider about the abnormal Docker pull times
    • Coordinate Kubernetes maintenance windows with LLM service owners so they do not clash with ongoing deployments
    • Build tooling to force a Ray cluster failover to the blue or green state instead of waiting for Ray to settle it
    View case
  2. IL-0234 An onboarding operator deployed without its scope file installed dev applications on other IONOS Kubernetes clusters
    IONOS Cloud2026
    Symptoms

    On 8 May 2026 IONOS Cloud reported "K8s Control Planes not available": some Managed Kubernetes customers in DE/TXL, DE/FRA and DE/FRA/2 had trouble connecting to the control plane and saw degraded Kubernetes functionality. Between 15:22 and 16:12 (UTC+2) applications meant for a new internal development cluster were installed on other customer clusters, which were also modified.

    Control plane unresponsiveImage pull failures
    Trigger

    A routine internal operation onboarded a new Managed Kubernetes development cluster. The scoping configuration file for the onboarding operator failed to reach the deployment branch because of a merge conflict, and at the same time a second change removed the operator from the exclusion list, which deployed it at once.

    Root cause
    Faulty deployment
    Component
    CI systemContainer registry and its credentials
    Contributing factor
    No configuration testsExcessive blast radiusMissed in review
    Remediation
    • Corrected the onboarding operator so it needs an explicit namespace configuration before acting on any cluster
    • Reconfigured the deployment system to fail deployments when required configuration files are missing
    • Rolled back the identified changes on the affected resources
    • Ran a recovery script, validated in staging, that removed all unintended installations and updates from the affected clusters
    Action items
    • Add a mandatory CI/CD step that confirms all required configuration files reached the target branch before any deployment starts
    • Require additional peer review and change management approval for every deployment that touches production, whatever its size
    • Give teams a way to inform each affected customer in time about the status of post-incident cleanup
    Lessons

    IONOS treats the unintended change to cluster configurations as a failure of the controls that should keep internal operations from affecting customer environments.

    View case
Check your cluster for this

These 2 cases involve CI system and Model inference server. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Faulty deployment2 cases

Component
CI system1Model inference server1Node pool1Container registry and its credentials1
Contributing factor
Excessive blast radius1No configuration tests1No staged rollout1Overlapping failures1Missed in review1
Organisation
IONOS Cloud1Lucidworks1
  1. IL-0237 A Kubernetes patch upgrade evicted every Ray Serve head and worker pod at once, and Lucidworks hosted LLMs went down
    Lucidworks2026
  2. IL-0234 An onboarding operator deployed without its scope file installed dev applications on other IONOS Kubernetes clusters
    IONOS Cloud2026