Incident Library
Applied filters Root causeCapacity exhaustionComponentPersistent volumes and stateful setsDomainKubernetesClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases4 cases

1 to 4 of 4

  1. IL-0067 Redis filled its persistent volume, corrupted its append-only file and crash-looped on every restart
    BC Government2025sev1
    Symptoms

    The Klamm production application was completely unavailable for about 30 minutes outside working hours. Klamm's queue workers timed out because they could not reach the Redis cache, and the services that depend on it were badly affected.

    Service outageContainer crash loopPersistent volume full
    Trigger

    A large, unexpected spike in I/O from a Klamm Laravel worker sharply increased requests to the Redis cache.

    Root cause
    Capacity exhaustion
    Component
    Persistent volumes and stateful setsIn-memory cache
    Contributing factor
    No capacity planning
    Remediation
    • backed off the Redis StatefulSet and the deployment
    • deleted the corrupted PVC and rebuilt it
    • restarted the StatefulSet and the deployment
    Action items
    • Increase the Klamm Redis cache volume to 100Mi
    • Integrate Sysdig monitoring on the namespace
    • Enable replica pods on the Redis cluster
    View case
  2. IL-0085 A control plane node of Magic Pages' Kubernetes cluster on Hetzner Cloud ran out of memory, the cluster's own calls then used up the Hetzner API rate limit, and all customer websites were down for about 7.5 hours
    Magic Pages2025
    Symptoms

    At around 09:33 UTC one of the three control plane nodes of the Kubernetes cluster that hosts Magic Pages went offline. Looking at the node showed that it had run out of memory.

    Service outageNode NotReadyCloud API rate limit exhausted
    Trigger

    One of the three control plane nodes ran out of memory and went offline at about 09:33 UTC; the attempts to replace it with hetzner-k3s failed while the cluster's own components were exhausting the Hetzner API rate limit.

    Root cause
    Capacity exhaustion
    Component
    Cloud controller managerPersistent volumes and stateful sets
    Contributing factor
    Unknown limitRetry stormMonitoring blind spotLimited debug accessSingle point of failureNo runbook
    Detection

    The founder found the control plane node offline at about 09:33 UTC. Monitoring alerts about unreachable customer websites started around the time the replacement attempts failed, and about half an hour of investigation identified the exhausted Hetzner API rate limit.

    Remediation
    • powered off all servers of the Kubernetes cluster to stop the API requests so the Hetzner rate limit could recover
    • waited until ratelimit-remaining had built up a buffer before starting recovery
    • restarted the control plane servers
    • started the essential Longhorn components by hand so Longhorn could heal and make storage volumes available
    • restarted the remaining websites by hand
    Action items
    • Monitor the Hetzner API call rate coming from the Kubernetes cluster nodes themselves to get early warning of high usage
    • Get a higher Hetzner API rate limit; Hetzner support was contacted
    • Investigate how to reduce the cluster's reliance on a single API, including alternatives to hetzner-k3s
    • Document and test a procedure for recovering Longhorn after a full cluster shutdown
    Lessons

    The running cluster, not only the deployment tooling, consumed the cloud provider's API quota, and in a failure it consumed it fast enough that the quota never recovered. The default limit of 3600 requests per hour was not enough for failure and recovery scenarios, and API usage from inside the cluster was not monitored. A tool that hides the provider's API errors slows diagnosis. After a full, non-graceful shutdown, Longhorn needed manual intervention to come back, which lengthened the outage.

    View case
  3. IL-0500 UToronto JupyterHub logins and server starts returned 500 errors when the hub's sqlite database disk filled up
    2i2c2022
    Symptoms

    On 31 October 2022 at 6:27 PM Pacific time, users of the UToronto JupyterHub run by 2i2c could no longer log in or start their servers, and the hub answered with 500 errors. Users who were already logged in could keep using the hub.

    High error ratePersistent volume full
    Trigger

    The JupyterHub log file, written to the same 10G disk as the hub's sqlite database, grew until the disk was completely full.

    Root cause
    Capacity exhaustion
    Component
    DatabasePersistent volumes and stateful setsJupyterHub
    Contributing factor
    No alert on limit
    Detection

    A user emailed 2i2c support at 7:13 PM that users could not log in; the team acknowledged the issue at 8:33 PM. There was no alert for the problem.

    Remediation
    • Compressed the hub log file into /tmp from a shell in the hub pod, because it could not be compressed in place on the full disk
    • Copied the compressed log to a local machine so the hub pod could be restarted without losing it
    • Resized the hub database disk (the hub-db-dir PVC) from 10G to 60G
    • Restarted the hub pod, which came back with the larger disk and restored service
    Action items
    • Stop keeping hub logs on the same disk as the hub database
    • Add a Grafana graph of free space on the JupyterHub database disk
    • Alert on elevated error rates
    Lessons

    Kubernetes dynamic volume resizing let the hub database disk grow from 10G to 60G without much effort. The risk of keeping logs on the database disk had been identified during the earlier migration, but the team had no solid process for following it through, and because no alert existed the team had to wait for a person to report the outage.

    View case
  4. IL-0518 gitlab.freedesktop.org on GKE ran short of volume space and a planned 30-minute storage upgrade became a 4-hour outage
    freedesktop.org2018
    Symptoms

    Shortly after 9am UTC on 17 July 2018, the disk space of the GitLab cluster behind gitlab.freedesktop.org was critical, and some API requests were failing because of the shortage. The administrator first learned of the failure when people mentioned it on IRC.

    Persistent volume fullHigh error rateService outage
    Trigger

    Many recently created forks of Mesa, each about 2.1GB, filled the disk space of the GitLab cluster.

    Root cause
    Capacity exhaustion
    Component
    Persistent volumes and stateful setsCloud block storageHelm
    Contributing factor
    Unverified assumptionUnrepresentative stagingMonitoring blind spotChange during peak hours
    Detection

    People mentioned the failure on IRC. Monitoring and alerting on disk usage and API error rate had not been set up yet.

    Remediation
    • Dropped all inbound traffic to GitLab at the firewall except from the administrator's IP
    • Took a manual backup by streaming tar over SSH to an encrypted partition on a local machine
    • Snapshotted all GCE disks behind the PersistentVolumes
    • Copied the data from the old disk snapshots to new enlarged volumes through a temporary GCE VM
    • Re-did the copy into new empty volumes with the correct directory structure and checked repositories, users and workflows
    • Restored public access to GitLab
    Action items
    • Set up monitoring, logging and alerts
    • Find ways to communicate outages much better; this one was announced only on IRC, and the lists were not told when it dragged on
    • Use the Cloud console clients instead of the slow Cloud Console web UI
    • Add more GitLab administrators, ideally someone familiar with Kubernetes deployments
    • Move from the single Omnibus container to GitLab's cloud native deployment with one container per service
    • Move storage to resizeable volumes or to Google Cloud Storage
    Lessons

    The backup step alone used up the outage window; backups had grown along with the disk problem, and a test run before announcing the window could have measured it. The changes had been tested on a scratch cluster, but not exactly as run, and tests of steps such as disk snapshots could have been more realistic. A single GitLab administrator who was new to Kubernetes was a single point of failure. Keeping at least three copies of the data from the first destructive step meant no data was lost.

    View case
Check your cluster for this

These 4 cases involve Persistent volumes and stateful sets and Cloud block storage. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion4 cases

Component
Persistent volumes and stateful sets4Cloud block storage1In-memory cache1Cloud controller manager1Database1Helm1JupyterHub1
Contributing factor
Monitoring blind spot2Unverified assumption1Change during peak hours1Unknown limit1Limited debug access1No alert on limit1No capacity planning1No runbook1Retry storm1Single point of failure1Unrepresentative staging1
Organisation
2i2c1BC Government1freedesktop.org1Magic Pages1
  1. IL-0067 Redis filled its persistent volume, corrupted its append-only file and crash-looped on every restart
    BC Government2025
  2. IL-0085 A control plane node of Magic Pages' Kubernetes cluster on Hetzner Cloud ran out of memory, the cluster's own calls then used up the Hetzner API rate limit, and all customer websites were down for about 7.5 hours
    Magic Pages2025
  3. IL-0500 UToronto JupyterHub logins and server starts returned 500 errors when the hub's sqlite database disk filled up
    2i2c2022
  4. IL-0518 gitlab.freedesktop.org on GKE ran short of volume space and a planned 30-minute storage upgrade became a 4-hour outage
    freedesktop.org2018