Cases16 cases

4 to 8 of 16

1. IL-0067 Redis filled its persistent volume, corrupted its append-only file and crash-looped on every restart

   BC Government2025sev1

   Symptoms

   The Klamm production application was completely unavailable for about 30 minutes outside working hours. Klamm's queue workers timed out because they could not reach the Redis cache, and the services that depend on it were badly affected.

   Service outageContainer crash loopPersistent volume full

   Trigger

   A large, unexpected spike in I/O from a Klamm Laravel worker sharply increased requests to the Redis cache.

   Root cause

   Capacity exhaustion

   Component

   Persistent volumes and stateful setsIn-memory cache

   Contributing factor

   No capacity planning

   Remediation

   • backed off the Redis StatefulSet and the deployment
   • deleted the corrupted PVC and rebuilt it
   • restarted the StatefulSet and the deployment

   Action items

   • Increase the Klamm Redis cache volume to 100Mi
   • Integrate Sysdig monitoring on the namespace
   • Enable replica pods on the Redis cluster

   View case
2. IL-0085 A control plane node of Magic Pages' Kubernetes cluster on Hetzner Cloud ran out of memory, the cluster's own calls then used up the Hetzner API rate limit, and all customer websites were down for about 7.5 hours

   Magic Pages2025

   Symptoms

   At around 09:33 UTC one of the three control plane nodes of the Kubernetes cluster that hosts Magic Pages went offline. Looking at the node showed that it had run out of memory.

   Service outageNode NotReadyCloud API rate limit exhausted

   Trigger

   One of the three control plane nodes ran out of memory and went offline at about 09:33 UTC; the attempts to replace it with hetzner-k3s failed while the cluster's own components were exhausting the Hetzner API rate limit.

   Root cause

   Capacity exhaustion

   Component

   Cloud controller managerPersistent volumes and stateful sets

   Contributing factor

   Unknown limitRetry stormMonitoring blind spotLimited debug accessSingle point of failureNo runbook

   Detection

   The founder found the control plane node offline at about 09:33 UTC. Monitoring alerts about unreachable customer websites started around the time the replacement attempts failed, and about half an hour of investigation identified the exhausted Hetzner API rate limit.

   Remediation

   • powered off all servers of the Kubernetes cluster to stop the API requests so the Hetzner rate limit could recover
   • waited until ratelimit-remaining had built up a buffer before starting recovery
   • restarted the control plane servers
   • started the essential Longhorn components by hand so Longhorn could heal and make storage volumes available
   • restarted the remaining websites by hand

   Action items

   • Monitor the Hetzner API call rate coming from the Kubernetes cluster nodes themselves to get early warning of high usage
   • Get a higher Hetzner API rate limit; Hetzner support was contacted
   • Investigate how to reduce the cluster's reliance on a single API, including alternatives to hetzner-k3s
   • Document and test a procedure for recovering Longhorn after a full cluster shutdown

   Lessons

   The running cluster, not only the deployment tooling, consumed the cloud provider's API quota, and in a failure it consumed it fast enough that the quota never recovered. The default limit of 3600 requests per hour was not enough for failure and recovery scenarios, and API usage from inside the cluster was not monitored. A tool that hides the provider's API errors slows diagnosis. After a full, non-graceful shutdown, Longhorn needed manual intervention to come back, which lengthened the outage.

   View case
3. IL-0489 Rejoining a failed node of Magic Pages' three-node MySQL cluster broke its metadata and took customer websites down

   Magic Pages2025

   Symptoms

   On 15 January 2025, from about 09:55 to 11:36 CET, a database outage at Magic Pages took down customer websites. It began when the founder tried to reconnect a failed third node to the high-availability MySQL cluster and the database server reported ambiguous errors about missing metadata information.

   Service outage

   Trigger

   One of the three nodes of the MySQL cluster went down, and reconnecting it failed with errors about missing metadata information.

   Root cause

   Dependency failure

   Component

   DatabasePersistent volumes and stateful sets

   Contributing factor

   Harmful manual actionSingle point of failureMonitoring blind spot

   Remediation

   • Restarted the storage server to fix PVC provisioning
   • Let the MySQL nodes restore and sync with each other for about 45 minutes to rebuild the cluster metadata
   • Cleaned up the corrupted cluster metadata
   • Redeployed MySQL Router

   Action items

   • Monitor the health of the storage system more closely
   • Improve the failover procedures for MySQL Router
   • Investigate and fix the status page so it shows reliable information

   Lessons

   The cluster survived the first node failure as designed and lost no data, but its metadata turned out to be a single point of failure. Better monitoring of PVC health could help prevent similar issues, the 45-minute recovery leaves room for improvement, and the status page has to detect failing customer sites reliably.

   View case
4. IL-0500 UToronto JupyterHub logins and server starts returned 500 errors when the hub's sqlite database disk filled up

   2i2c2022

   Symptoms

   On 31 October 2022 at 6:27 PM Pacific time, users of the UToronto JupyterHub run by 2i2c could no longer log in or start their servers, and the hub answered with 500 errors. Users who were already logged in could keep using the hub.

   High error ratePersistent volume full

   Trigger

   The JupyterHub log file, written to the same 10G disk as the hub's sqlite database, grew until the disk was completely full.

   Root cause

   Capacity exhaustion

   Component

   DatabasePersistent volumes and stateful setsJupyterHub

   Contributing factor

   No alert on limit

   Detection

   A user emailed 2i2c support at 7:13 PM that users could not log in; the team acknowledged the issue at 8:33 PM. There was no alert for the problem.

   Remediation

   • Compressed the hub log file into /tmp from a shell in the hub pod, because it could not be compressed in place on the full disk
   • Copied the compressed log to a local machine so the hub pod could be restarted without losing it
   • Resized the hub database disk (the hub-db-dir PVC) from 10G to 60G
   • Restarted the hub pod, which came back with the larger disk and restored service

   Action items

   • Stop keeping hub logs on the same disk as the hub database
   • Add a Grafana graph of free space on the JupyterHub database disk
   • Alert on elevated error rates

   Lessons

   Kubernetes dynamic volume resizing let the hub database disk grow from 10G to 60G without much effort. The risk of keeping logs on the database disk had been identified during the earlier migration, but the team had no solid process for following it through, and because no alert existed the team had to wait for a person to report the outage.

   View case
5. IL-0009 A duplicated generated name made Argo CD replace the InfluxDB Cloud production Application and delete the whole production environment

   InfluxData2022

   Symptoms

   Right after a pull request to the infrastructure-as-code repository was merged, every API endpoint of one InfluxDB Cloud production cluster began returning 404 errors. Writes and reads failed alike: no one could collect data, scheduled tasks failed to run and external queries did not work.

   Service outageResources deletedHigh error rate

   Trigger

   A merged pull request whose rendered YAML added a new Argo CD Application and AppProject that an automation error had named app01-cluster01, the name of the existing production Application, instead of app02-cluster01; reviewers did not notice the app01 and app02 difference.

   Root cause

   Configuration change

   Component

   Argo CDPersistent volumes and stateful setsConfiguration templating

   Contributing factor

   Missed in reviewNo runbookNo safeguard on destructive changeMonitoring blind spotAutomation amplification

   Detection

   Monitoring and alerting did not detect the issue right away, so the team's first response was to try to roll the change back rather than plan a recovery. The write-up does not say how the team found out.

   Remediation

   • stopped the revert of the merged pull request
   • restored the etcd metadata cluster
   • recreated the missing Kubernetes objects so the retained PersistentVolumes became available, then recreated the StatefulSets
   • restored the Zookeeper volumes from an hourly Velero backup
   • rebuilt the Kafka StatefulSet one pod at a time with health and readiness checks disabled
   • accepted writes into the Kafka write-ahead log and scaled up the write path with more replicas and higher memory requests and limits
   • recreated the storage pods, which downloaded data from object storage, reindexed it and replayed the write-ahead log
   • re-enabled Argo CD, queries, dashboards and scheduled tasks, restored replica counts and ran final QA checks

   Action items

   • Write runbooks for restoring state when Kubernetes objects such as PersistentVolumeClaims are deleted but the data on the underlying volumes is preserved
   • Set every volume in every environment to retain data when its PersistentVolumeClaim is deleted
   • Improve the process for handling public-facing incidents
   • Store generated YAML as one object per file, so an overwrite of an existing object shows up as such in review
   • Detect duplicate objects when generating YAML and warn before review, treating objects with the same group, namespace and name as the same even when the apiVersion string differs
   • Add the Argo CD Prune=false annotation to all stateful resources and to Namespace objects managed by Argo CD
   • Set FailOnSharedResource=true on Argo CD Applications so Argo CD fails before changing an object another Application manages or managed

   Lessons

   The biggest gap was that monitoring did not notice the deletion quickly, which pushed the team toward a reflex rollback; in a stateful system a revert can create a new, empty instance instead of bringing the old one back. Retained volumes and clients that retry writes are what kept data loss at zero. Rendering many objects into one file let an overwrite of an existing object look like an addition, so generated configuration needs one object per file and duplicate detection that compares objects the way Kubernetes does. Stateful resources need protection from pruning, and that protection must cover their Namespace too, or deleting the namespace removes everything inside it.

   View case

By root cause9