Cases3 cases

1 to 3 of 3

1. IL-0067 Redis filled its persistent volume, corrupted its append-only file and crash-looped on every restart

   BC Government2025sev1

   Symptoms

   The Klamm production application was completely unavailable for about 30 minutes outside working hours. Klamm's queue workers timed out because they could not reach the Redis cache, and the services that depend on it were badly affected.

   Service outageContainer crash loopPersistent volume full

   Trigger

   A large, unexpected spike in I/O from a Klamm Laravel worker sharply increased requests to the Redis cache.

   Root cause

   Capacity exhaustion

   Component

   Persistent volumes and stateful setsIn-memory cache

   Contributing factor

   No capacity planning

   Remediation

   • backed off the Redis StatefulSet and the deployment
   • deleted the corrupted PVC and rebuilt it
   • restarted the StatefulSet and the deployment

   Action items

   • Increase the Klamm Redis cache volume to 100Mi
   • Integrate Sysdig monitoring on the namespace
   • Enable replica pods on the Redis cluster

   View case
2. IL-0500 UToronto JupyterHub logins and server starts returned 500 errors when the hub's sqlite database disk filled up

   2i2c2022

   Symptoms

   On 31 October 2022 at 6:27 PM Pacific time, users of the UToronto JupyterHub run by 2i2c could no longer log in or start their servers, and the hub answered with 500 errors. Users who were already logged in could keep using the hub.

   High error ratePersistent volume full

   Trigger

   The JupyterHub log file, written to the same 10G disk as the hub's sqlite database, grew until the disk was completely full.

   Root cause

   Capacity exhaustion

   Component

   DatabasePersistent volumes and stateful setsJupyterHub

   Contributing factor

   No alert on limit

   Detection

   A user emailed 2i2c support at 7:13 PM that users could not log in; the team acknowledged the issue at 8:33 PM. There was no alert for the problem.

   Remediation

   • Compressed the hub log file into /tmp from a shell in the hub pod, because it could not be compressed in place on the full disk
   • Copied the compressed log to a local machine so the hub pod could be restarted without losing it
   • Resized the hub database disk (the hub-db-dir PVC) from 10G to 60G
   • Restarted the hub pod, which came back with the larger disk and restored service

   Action items

   • Stop keeping hub logs on the same disk as the hub database
   • Add a Grafana graph of free space on the JupyterHub database disk
   • Alert on elevated error rates

   Lessons

   Kubernetes dynamic volume resizing let the hub database disk grow from 10G to 60G without much effort. The risk of keeping logs on the database disk had been identified during the earlier migration, but the team had no solid process for following it through, and because no alert existed the team had to wait for a person to report the outage.

   View case
3. IL-0518 gitlab.freedesktop.org on GKE ran short of volume space and a planned 30-minute storage upgrade became a 4-hour outage

   freedesktop.org2018

   Symptoms

   Shortly after 9am UTC on 17 July 2018, the disk space of the GitLab cluster behind gitlab.freedesktop.org was critical, and some API requests were failing because of the shortage. The administrator first learned of the failure when people mentioned it on IRC.

   Persistent volume fullHigh error rateService outage

   Trigger

   Many recently created forks of Mesa, each about 2.1GB, filled the disk space of the GitLab cluster.

   Root cause

   Capacity exhaustion

   Component

   Persistent volumes and stateful setsCloud block storageHelm

   Contributing factor

   Unverified assumptionUnrepresentative stagingMonitoring blind spotChange during peak hours

   Detection

   People mentioned the failure on IRC. Monitoring and alerting on disk usage and API error rate had not been set up yet.

   Remediation

   • Dropped all inbound traffic to GitLab at the firewall except from the administrator's IP
   • Took a manual backup by streaming tar over SSH to an encrypted partition on a local machine
   • Snapshotted all GCE disks behind the PersistentVolumes
   • Copied the data from the old disk snapshots to new enlarged volumes through a temporary GCE VM
   • Re-did the copy into new empty volumes with the correct directory structure and checked repositories, users and workflows
   • Restored public access to GitLab

   Action items

   • Set up monitoring, logging and alerts
   • Find ways to communicate outages much better; this one was announced only on IRC, and the lists were not told when it dragged on
   • Use the Cloud console clients instead of the slow Cloud Console web UI
   • Add more GitLab administrators, ideally someone familiar with Kubernetes deployments
   • Move from the single Omnibus container to GitLab's cloud native deployment with one container per service
   • Move storage to resizeable volumes or to Google Cloud Storage

   Lessons

   The backup step alone used up the outage window; backups had grown along with the disk problem, and a test run before announcing the window could have measured it. The changes had been tested on a scratch cluster, but not exactly as run, and tests of steps such as disk snapshots could have been more realistic. A single GitLab administrator who was new to Kubernetes was a single point of failure. Keeping at least three copies of the data from the first destructive step meant no data was lost.

   View case

By root cause1