Cases14 cases

2 to 6 of 14

1. IL-0237 A Kubernetes patch upgrade evicted every Ray Serve head and worker pod at once, and Lucidworks hosted LLMs went down

   Lucidworks2026sev1

   Symptoms

   From 16:18 UTC on 28 May 2026 the Lucidworks AI hosted LLMs in the us-southcarolina region were unavailable: queries to llama-3-8b-instruct, llama-3v2-3b-instruct and phi-4-multimodal-instruct returned 500 or 429 errors. Search, embedding models, the Lucidworks Platform UI and passthrough LLM calls kept working.

   Service outagePod evictionImage pull failures

   Trigger

   A routine Kubernetes patch upgrade on a cluster in us-southcarolina cycled its node pools one after another in quick succession.

   Root cause

   Faulty deployment

   Component

   Node poolModel inference server

   Contributing factor

   No staged rolloutOverlapping failures

   Remediation

   • Deleted the degraded replacement head node that could not pull container images
   • Waited for the LLM image pulls to finish on the replacement nodes
   • Launched new nodes with the required LLM images and redirected Lucidworks AI traffic to them

   Action items

   • Upgrade clusters that host LLM workloads in sequence, with each node pool fully healthy before the next one is upgraded
   • Investigate image pre-loading, such as pre-baked disks or image streaming, for large ML model images
   • Open a support ticket with the cloud provider about the abnormal Docker pull times
   • Coordinate Kubernetes maintenance windows with LLM service owners so they do not clash with ongoing deployments
   • Build tooling to force a Ray cluster failover to the blue or green state instead of waiting for Ray to settle it

   View case
2. IL-0542 Twingate authentication and authorization requests failed at higher rates after traffic moved onto a smaller GKE cluster

   Twingate2025

   Symptoms

   On 21 March 2025, between 21:10 and 21:41 UTC, part of the authentication and authorization requests to Twingate services failed with elevated error rates.

   High error rate

   Trigger

   Twingate saw anomalies in one of its GKE clusters and scaled down the deployments in that cluster to protect the rest of the system, which moved its traffic to the other clusters.

   Root cause

   Cloud provider outage

   Component

   Cloud load balancerGKE control planeHorizontal pod autoscalerCluster autoscaler

   Contributing factor

   Retry stormNo headroomOverlapping failuresHarmful manual action

   Detection

   Twingate observed anomalies in one of its GKE clusters at the start of the incident.

   Remediation

   • Reintroduced capacity in the cluster that had been scaled down, so traffic was rebalanced across regions

   Action items

   • Keep working with the cloud provider to find the cause of the unexpected timeouts and of the cluster update's impact
   • Run deployments in all regions with the same number of replicas in the HPA configuration
   • Raise the maximum node counts of the cluster node autoscalers to leave more room to scale up
   • Configure the HPA to scale up faster

   View case
3. IL-0084 Packet drops on a bare metal provider's vSwitch forced Namespace to rebuild its fra3 region and left global capacity short, and the next day slow NVMe writes on a control plane node drove the Kubernetes API server to very high CPU

   Namespace2024

   Symptoms

   From 17 October 2024, packets through the vSwitch that joined Namespace's bare metal machines at one provider in the fra3 region were dropped at random. Retries in TCP, QUIC and the Namespace control plane hid most of it, but the SLO error budget for latency targets on key interactions began to burn.

   Intermittent connection failuresHigh latencyControl plane lagService outage

   Trigger

   Random packet drops on the bare metal provider's vSwitch from 17 October 2024; for the second outage, a rate of Kubernetes changes that the slow NVMe on one control plane node could no longer keep up with.

   Root cause

   Cloud provider outage

   Component

   Kubernetes API server

   Contributing factor

   No capacity planningNo headroomOverlapping failures

   Detection

   SLO-based monitoring of latency targets showed the error budget burning while retries hid the individual packet drops.

   Remediation

   • diverted all customer traffic from fra3 to other regions
   • connected each provider machine to the zrh1 cluster over Wireguard instead of the vSwitch
   • repaved every fra3 machine with a new identity, certificates and encryption-at-rest keys, dropping regional caches to recover faster
   • cloned existing datasets in a more distributed way to speed up repaving the machines
   • added machines back to the new region in batches and returned the region to the compute pool
   • redirected workloads away from zrh1 during the second outage
   • cloned the failing control plane node's key state to a higher-capacity node that took over its role and IP address

   Action items

   • Credit all instances that ran during the outages back to customers
   • Build up backup regions and solutions so a similar event has options that do not affect customers

   Lessons

   Running hot with a delayed hardware delivery left no room to take a region out, so fixing a provider network fault turned into a global capacity shortage. After one outage the team was anchored on its cause and first looked for network problems in the second outage, which had a different cause: a control plane node whose disk could not keep up with the Kubernetes change rate.

   View case
4. IL-0114 An unpinned AWS VPC CNI plugin was updated automatically during a capacity expansion, leaving new pods Pending without IP addresses and stopping CoW Protocol settlements for about 3.5 hours

   CoW Protocol2024

   Symptoms

   At 16:40 UTC on 19 April 2024 CoW Protocol's mainnet production environment stopped producing settlements; the outage officially began then.

   Pods PendingIP address exhaustionPod evictionService outage

   Trigger

   During a cluster capacity expansion, the Infrastructure as Code automation updated the AWS VPC CNI plugin, because the CNI version was not hardcoded in the IaC and the IaC library version had recently changed.

   Root cause

   Configuration change

   Component

   AWS VPC CNINode pool

   Contributing factor

   Unpinned versionOverlapping failures

   Detection

   The CNI version mismatch was identified as the cause at 17:03 UTC, 23 minutes after settlements stopped.

   Remediation

   • upgraded the Kubernetes cluster
   • set up new node groups with a new version of the networking plugin

   Lessons

   Versions in IaC scripts should be specified explicitly so an automation run cannot update a component such as the CNI plugin unintentionally. Changes in production need change management that is easy to revert.

   View case
5. IL-0386 After a move to Google Cloud, Postgres connection timeouts and memcached i/o timeouts spiked on GKE nodes

   incident.io2023

   Symptoms

   Shortly after incident.io moved its infrastructure fully into Google Cloud, the application saw a larger-than-normal number of Postgres connection timeouts, and new database connections per second were in the hundreds, spiking up and down. At the same time, the memcached instance used for transient data and faster API responses had sporadic connection timeouts.

   Intermittent timeoutsPacket loss

   Trigger

   After the move to Google Cloud, the application opened hundreds of new Postgres connections per second, and an accidental database join made it send thousands of duplicate calls to an external third party in parallel.

   Root cause

   Capacity exhaustion

   Component

   CNI pluginDatabaseIn-memory cache

   Contributing factor

   Unreviewed defaultsOverlapping failures

   Detection

   An incident was declared when the number of Postgres connection timeouts rose above normal. The remaining cache timeouts were traced through the logs of the Kubernetes node that served an affected request, the request's trace and the node's metrics.

   Remediation

   • Doubled the maximum Postgres connection lifespan from 15m to 30m
   • Made the Postgres connection pools static so they always keep N connections open
   • Raised the memcache client's MaxIdleConns from 2 to 20
   • Moved temporarily from Google's hosted memcached to a memcached inside the Kubernetes cluster
   • Shipped the fix for the accidental database join that made thousands of duplicate external calls

   Action items

   • Set more sensible concurrency limits for outbound network calls

   Lessons

   Network timeouts are not always local to one request or pod: when they appear, the whole activity of the node, including other pods on it, needs to be checked. Staying within a provider's rate limits does not keep outbound calls from affecting a node, so outbound network calls need more sensible concurrency limits.

   View case

By root cause8