Incident Library

IL-0518 freedesktop.org · 2018-07-17kubernetesgcp

gitlab.freedesktop.org on GKE ran short of volume space and a planned 30-minute storage upgrade became a 4-hour outage

gitlab.freedesktop.org ran on Google Kubernetes Engine with Helm charts and Kubernetes 1.10.x, with the whole service in a single pod running the GitLab CE image (11.0.4 at the time). The Git repositories lived on a data partition exposed through a Kubernetes PersistentVolumeClaim, and other data in PostgreSQL on another PVC. Kubernetes persistent volumes were not resizeable, with only alpha support in 1.10.x. The daily backup collects data into a local directory and then writes an uncompressed tarball there, so it needs a bit over twice the backup size in free space. When forks of Mesa filled the disk on 17 July 2018, the administrator announced urgent downtime, tested migrations on a scratch cluster, and announced a window from 2 to 2:30pm UTC, a time when Europe and the American east coast were active.

At 2pm UTC the administrator blocked all inbound traffic except from one IP, took a manual backup by streaming tar over SSH to an encrypted partition on a local machine, because writing the tarball locally would have filled the disk, and snapshotted every GCE disk behind the PersistentVolumes in the Cloud Console web UI. Both took far longer than planned, because the backups had grown and the web UI was slow, and the window was extended to 3pm. Resizing the volumes then failed: a static-sized disk could not be made resizeable without recreating the PersistentVolumes, and GKE did not allow enabling the alpha resize feature. Restoring snapshots also failed, because a snapshot cannot be restored into a live disk directly, and creating a Kubernetes PV/PVC from an existing GCE disk erased its content. Helm blocked indefinitely when a resource was missing because of a typo, and the administrator's own network connection degraded.

In the end the administrator stopped the Kubernetes services, attached the new enlarged volumes and the old disk snapshots to a temporary GCE VM and copied the data across. The first copy did not keep the subdirectory that Kubernetes mounts as the MySQL data volume, so the database looked empty. The administrator destroyed the resources, re-did the copy into new empty volumes with the right directory structure, checked repositories, users and typical workflows, and restored public access around 18:45 UTC. The post gives the full outage as 14:00 to 18:00 UTC.

Severity
–
Scope
The GitLab web UI, API and HTTPS git clones on gitlab.freedesktop.org were completely unavailable; anongit and cgit kept working
Users affected
–
Region
–
Time to detect
–
Time to resolve
4h 45m

Symptoms

Shortly after 9am UTC on 17 July 2018, the disk space of the GitLab cluster behind gitlab.freedesktop.org was critical, and some API requests were failing because of the shortage. The administrator first learned of the failure when people mentioned it on IRC.

A large number of recently created forks of Mesa, each needing ~2.1GB because a fork is a plain git clone with no shared objects, had filled the storage. The daily backup cron job was failing for the same reason, so there were no current backups, and one more Mesa or kernel fork could have used up the remaining disk space.

During the storage upgrade the GitLab web UI, the API and HTTPS git clones through https://gitlab.freedesktop.org were completely unavailable and gave connection timeout errors, because inbound traffic was dropped at the firewall. anongit and cgit kept working, and no data was lost.

Symptom Persistent volume fullHigh error rateService outage

Root cause

Root cause Capacity exhaustion

Component Persistent volumes and stateful setsCloud block storageHelm
Contributing factor Unverified assumptionUnrepresentative stagingMonitoring blind spotChange during peak hours
Trigger
Many recently created forks of Mesa, each about 2.1GB, filled the disk space of the GitLab cluster.

Detection

People mentioned the failure on IRC. Monitoring and alerting on disk usage and API error rate had not been set up yet.

Remediation

Action items

Lessons

The backup step alone used up the outage window; backups had grown along with the disk problem, and a test run before announcing the window could have measured it. The changes had been tested on a scratch cluster, but not exactly as run, and tests of steps such as disk snapshots could have been more realistic. A single GitLab administrator who was new to Kubernetes was a single point of failure. Keeping at least three copies of the data from the first destructive step meant no data was lost.

Sources

References

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Persistent volumes and stateful sets, Cloud block storage, Helm among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.