Cases3 cases

1 to 3 of 3

1. IL-0063 VPN outage after recreating a Redis StatefulSet restarted the whole Redis cluster without readiness checks

   AdGuard VPN2025

   Symptoms

   The day began with an alert that several instances of the Redis cluster were running out of disk space. That part was fixed quickly. The outage came an hour later: right after the Redis StatefulSet was deleted and recreated at 8:10 UTC, Kubernetes restarted every Redis pod within about five minutes, and problems started to appear.

   Service outageOOMKilledProbe failuresHigh latency

   Trigger

   To make Git match the volumes that had been resized by hand, the Redis StatefulSet was deleted with orphan (non-cascading) deletion and recreated with the new storage template; Kubernetes reconciled the new spec by restarting the entire Redis cluster.

   Root cause

   Configuration change

   Component

   Persistent volumes and stateful setsIn-memory cacheContainer memory limit

   Contributing factor

   No health probesUnverified assumptionSingle point of failureRetry stormNo headroom

   Detection

   An alert about Redis instances running out of disk space started the work; the outage itself showed up as backend errors about Redis connections right after the StatefulSet was recreated.

   Remediation

   • made the Redis masters forget obsolete node identities, reset the affected replicas and rejoined them, then waited for replication to catch up
   • identified the component that was actually running out of memory, scaled the backend service and raised memory limits for that component
   • temporarily relaxed probes so backend instances took traffic immediately instead of flapping between ready and not ready
   • reverted the temporary workarounds at 10:00 UTC

   Action items

   • Add startup probes so a Redis pod is not marked ready until its dataset is loaded
   • Build an own control plane for Redis that handles restarted pods stuck to an old master address
   • Reduce dependence on the single Redis cluster: move some important data to other stores, such as a separate Redis cluster per server, and use Kafka for some backend components

   Lessons

   The disk resize was not the problem; the cleanup afterwards was. An immutable field pushes operators toward deleting and recreating the controller, and that can set off a full reconcile. Startup and readiness probes are what slow a rolling restart down to a safe pace; without them a distributed store restarts in the most fragile way. Recovery plans must expect a cluster to heal slowly, and dependent services need to degrade gracefully while it does.

   View case
2. IL-0034 A crashing stats agent sidecar marked every pod unready and the service mesh pulled all services out of traffic

   Airbnb2019

   Symptoms

   An operation to upgrade the stats agent in Kubernetes was announced in advance, with a warning that metrics might be unavailable to services for a few minutes. A few minutes later a reply came in saying that all of the sender's Kubernetes services seemed to be down.

   Probe failuresService outage

   Trigger

   A scheduled upgrade of the stats agent sidecar, during which the agent container crashed.

   Root cause

   Faulty deployment

   Component

   Telemetry agentService discovery

   Contributing factor

   Unverified assumptionExcessive blast radius

   Detection

   A reply to the operation announcement, a few minutes later, said that all of the sender's Kubernetes services appeared to be down.

   Action items

   • Mark logging, tracing and stats agent containers as non-critical and leave them out of the service mesh's pod readiness check

   Lessons

   Every container in a pod affects the pod's readiness, so each sidecar needs a deliberate decision about how its health should count. A sidecar that crashes is not ready even when it has no probes, and any logic that requires all containers to be ready will take the pod out of traffic.

   View case
3. IL-0054 All ingress traffic lost while draining a GKE node pool because the load balancer only targeted new nodes without ingress-nginx pods

   DevOps Hof2019

   Symptoms

   While the old nodes of a GKE cluster were being drained to move workloads to a newly created node pool, none of the services in the cluster could be reached through ingress-nginx any more.

   Service outageProbe failures

   Trigger

   Cordoning and draining the old node pool removed those nodes from the GCP load balancer backend while the ingress-nginx pods were still running on them.

   Root cause

   Operator error

   Component

   Ingress controllerkube-proxyNode pool

   Contributing factor

   Unreviewed defaultsUnverified assumption

   Detection

   The team noticed during the drain that no service in the cluster was reachable through ingress-nginx.

   Remediation

   • deleted the nginx-ingress-controller pods still running on the old nodes so they were rescheduled on the new, ready nodes

   Lessons

   With externalTrafficPolicy: Local, the load balancer can only reach nodes that run an ingress pod, and cordoned nodes leave the load balancer backend immediately; if the ingress pods stay behind on cordoned nodes, all ingress traffic stops. Before any change that might touch traffic, look twice into the documentation.

   View case

By root cause3