Cases5 cases

1 to 5 of 5

1. IL-0063 VPN outage after recreating a Redis StatefulSet restarted the whole Redis cluster without readiness checks

   AdGuard VPN2025

   Symptoms

   The day began with an alert that several instances of the Redis cluster were running out of disk space. That part was fixed quickly. The outage came an hour later: right after the Redis StatefulSet was deleted and recreated at 8:10 UTC, Kubernetes restarted every Redis pod within about five minutes, and problems started to appear.

   Service outageOOMKilledProbe failuresHigh latency

   Trigger

   To make Git match the volumes that had been resized by hand, the Redis StatefulSet was deleted with orphan (non-cascading) deletion and recreated with the new storage template; Kubernetes reconciled the new spec by restarting the entire Redis cluster.

   Root cause

   Configuration change

   Component

   Persistent volumes and stateful setsIn-memory cacheContainer memory limit

   Contributing factor

   No health probesUnverified assumptionSingle point of failureRetry stormNo headroom

   Detection

   An alert about Redis instances running out of disk space started the work; the outage itself showed up as backend errors about Redis connections right after the StatefulSet was recreated.

   Remediation

   • made the Redis masters forget obsolete node identities, reset the affected replicas and rejoined them, then waited for replication to catch up
   • identified the component that was actually running out of memory, scaled the backend service and raised memory limits for that component
   • temporarily relaxed probes so backend instances took traffic immediately instead of flapping between ready and not ready
   • reverted the temporary workarounds at 10:00 UTC

   Action items

   • Add startup probes so a Redis pod is not marked ready until its dataset is loaded
   • Build an own control plane for Redis that handles restarted pods stuck to an old master address
   • Reduce dependence on the single Redis cluster: move some important data to other stores, such as a separate Redis cluster per server, and use Kafka for some backend components

   Lessons

   The disk resize was not the problem; the cleanup afterwards was. An immutable field pushes operators toward deleting and recreating the controller, and that can set off a full reconcile. Startup and readiness probes are what slow a rolling restart down to a safe pace; without them a distributed store restarts in the most fragile way. Recovery plans must expect a cluster to heal slowly, and dependent services need to degrade gracefully while it does.

   View case
2. IL-0101 After Google forced a GKE control plane upgrade to 1.33.2 at a deprecation deadline, no new node could initialise, so autoscaling and spot node replacement stopped for 15 hours

   Wealthy2025sev1

   Symptoms

   Nodes that already existed kept running, but every new node, spot or standard, on either v1.32 or v1.33, failed to initialise and could not join the cluster. Spot nodes that were preempted and recreated never came back, so services degraded progressively: background workers on spot instances, reporting services and order allocation stopped completely, and the APIs on existing nodes kept answering but could not scale for load spikes.

   Scale-up failureContainer crash loopService outageDNS resolution failures

   Trigger

   Google upgraded the GKE control plane from v1.32.x to v1.33.2 at the version deprecation deadline, although auto-upgrade was disabled on the node pools.

   Root cause

   Breaking change in an upgrade

   Component

   GKE control planeNode poolClient-side DNS resolver and cachePersistent volumes and stateful setsConfigMaps and Secrets

   Contributing factor

   Deferred upgradesSingle point of failureIneffective rollbackNo headroom

   Remediation

   • Created a new GKE cluster at v1.33, the version of the forced upgrade
   • Upgraded Istio to 1.27.1 for Kubernetes 1.33 compatibility
   • Updated the cert-manager configuration for Kubernetes 1.33
   • Migrated all production workloads to the new cluster and verified pod scheduling, DNS resolution, PVC mounting and external integrations
   • Decommissioned the broken cluster

   Action items

   • Configure all clusters to use the GKE stable release channel
   • Implement a separate management cluster for Argo CD, monitoring and other control components (Q4 2025)
   • Review spot instance usage for critical workloads and add on-demand fallback pools
   • Document the SRE upgrade procedure: management cluster first, production after a monitoring period of several days
   • Document a Kubernetes, Istio and cert-manager compatibility matrix
   • Create an automated upgrade testing pipeline
   • Reserve static IPs for quick migrations
   • Pin versions of Istio, cert-manager, CSI and DNS addons
   • Stay current with stable releases and regularly check GKE alerts and announcements to avoid deprecation deadlines

   Lessons

   GKE upgrades the control plane at a version's deprecation deadline even when auto-upgrade is disabled, so staying inside the supported version window is how to avoid a forced upgrade. A control plane upgrade can leave existing nodes healthy while every new node fails, and spot nodes that cannot rejoin after preemption turn this into cascading failures. Relying heavily on spot instances for critical workers turned a node join failure into a progressive outage. Kubernetes upgrades have to be planned together with dependent components such as Istio and cert-manager, and without a standby cluster on the same provider there was no immediate failover; the only recovery left was building a new cluster.

   View case
3. IL-0085 A control plane node of Magic Pages' Kubernetes cluster on Hetzner Cloud ran out of memory, the cluster's own calls then used up the Hetzner API rate limit, and all customer websites were down for about 7.5 hours

   Magic Pages2025

   Symptoms

   At around 09:33 UTC one of the three control plane nodes of the Kubernetes cluster that hosts Magic Pages went offline. Looking at the node showed that it had run out of memory.

   Service outageNode NotReadyCloud API rate limit exhausted

   Trigger

   One of the three control plane nodes ran out of memory and went offline at about 09:33 UTC; the attempts to replace it with hetzner-k3s failed while the cluster's own components were exhausting the Hetzner API rate limit.

   Root cause

   Capacity exhaustion

   Component

   Cloud controller managerPersistent volumes and stateful sets

   Contributing factor

   Unknown limitRetry stormMonitoring blind spotLimited debug accessSingle point of failureNo runbook

   Detection

   The founder found the control plane node offline at about 09:33 UTC. Monitoring alerts about unreachable customer websites started around the time the replacement attempts failed, and about half an hour of investigation identified the exhausted Hetzner API rate limit.

   Remediation

   • powered off all servers of the Kubernetes cluster to stop the API requests so the Hetzner rate limit could recover
   • waited until ratelimit-remaining had built up a buffer before starting recovery
   • restarted the control plane servers
   • started the essential Longhorn components by hand so Longhorn could heal and make storage volumes available
   • restarted the remaining websites by hand

   Action items

   • Monitor the Hetzner API call rate coming from the Kubernetes cluster nodes themselves to get early warning of high usage
   • Get a higher Hetzner API rate limit; Hetzner support was contacted
   • Investigate how to reduce the cluster's reliance on a single API, including alternatives to hetzner-k3s
   • Document and test a procedure for recovering Longhorn after a full cluster shutdown

   Lessons

   The running cluster, not only the deployment tooling, consumed the cloud provider's API quota, and in a failure it consumed it fast enough that the quota never recovered. The default limit of 3600 requests per hour was not enough for failure and recovery scenarios, and API usage from inside the cluster was not monitored. A tool that hides the provider's API errors slows diagnosis. After a full, non-graceful shutdown, Longhorn needed manual intervention to come back, which lengthened the outage.

   View case
4. IL-0489 Rejoining a failed node of Magic Pages' three-node MySQL cluster broke its metadata and took customer websites down

   Magic Pages2025

   Symptoms

   On 15 January 2025, from about 09:55 to 11:36 CET, a database outage at Magic Pages took down customer websites. It began when the founder tried to reconnect a failed third node to the high-availability MySQL cluster and the database server reported ambiguous errors about missing metadata information.

   Service outage

   Trigger

   One of the three nodes of the MySQL cluster went down, and reconnecting it failed with errors about missing metadata information.

   Root cause

   Dependency failure

   Component

   DatabasePersistent volumes and stateful sets

   Contributing factor

   Harmful manual actionSingle point of failureMonitoring blind spot

   Remediation

   • Restarted the storage server to fix PVC provisioning
   • Let the MySQL nodes restore and sync with each other for about 45 minutes to rebuild the cluster metadata
   • Cleaned up the corrupted cluster metadata
   • Redeployed MySQL Router

   Action items

   • Monitor the health of the storage system more closely
   • Improve the failover procedures for MySQL Router
   • Investigate and fix the status page so it shows reliable information

   Lessons

   The cluster survived the first node failure as designed and lost no data, but its metadata turned out to be a single point of failure. Better monitoring of PVC health could help prevent similar issues, the 45-minute recovery leaves room for improvement, and the status page has to detect failing customer sites reliably.

   View case
5. IL-0196 Adding a second Redis replica and lowering memory limits in one RedisFailover change restarted the only Redis pod that held data, and Sentinel promoted the empty new replica, erasing the data

   Palark2020

   Symptoms

   Right after an updated RedisFailover resource was rolled out, the large Redis database lost all its data and the service went down; the data had to be restored from a backup taken just before the change.

   Data lossService outage

   Trigger

   A RedisFailover change that raised the number of Redis replicas from one to two and, in the same rollout, lowered the containers' memory requests and limits from 22 GB to 19 GB.

   Root cause

   Software defect

   Component

   DatabasePersistent volumes and stateful setsContainer memory limit

   Contributing factor

   No staged rolloutSingle point of failureUnverified assumption

   Remediation

   • restored the Redis data from the backup prepared before the rollout

   Action items

   • Report the readiness probe flaw to the Redis Operator project and test the developers' fix
   • Reconsider the use of the Spotahome Redis Operator
   • Clean up the Redis databases and fix the application code that created the excess keys
   • Consider Redis built-in clustering and sharding

   Lessons

   Changing the replica count and the pod resources in one rollout turned a seemingly harmless change into a rolling restart of the only pod with data while the new replica was still empty. A readiness probe that only checks that Redis answers a ping does not tell whether a replica has finished syncing. Operators for critical stateful components need to be evaluated for maturity before production use, and the dump.rdb file should be backed up before any change to the Redis cluster.

   View case

By root cause5