Cases3 cases

1 to 3 of 3

1. IL-0321 GCP snapshots on all Patroni nodes stalled GitLab.com's primary PostgreSQL disk, Patroni failed over and 5xx spiked

   GitLab2021

   Symptoms

   At 23:38 UTC on 2021-01-16 GitLab.com error rates spiked, and a minute later multiple alerts for service disruption arrived, two minutes after GCP disk snapshots had been started on all Patroni database nodes. Some requests that needed database writes returned 5xx errors for about four to five minutes, while page refreshes checked during the incident responded normally.

   High error rate

   Trigger

   During the change production#2648, at 23:36 UTC an engineer started GCP disk snapshots on all Patroni nodes at the same time, as a safety step before a change to the filesystem, believing the snapshot was a transparent background operation through the GCP API.

   Root cause

   Capacity exhaustion

   Component

   DatabaseDatabase failover managerCloud block storage

   Contributing factor

   Unverified assumptionNo staged rollout

   Detection

   A flood of alerts about downstream impact; multiple service disruption alerts arrived at 23:39 UTC, a minute after the error spikes began. The incident was declared in Slack at 23:56.

   Remediation

   • Patroni invalidated patroni-06's lock and promoted patroni-03 as the new primary
   • Cancelled the snapshot requests in two of the three zones

   Action items

   • Update the Patroni and PostgreSQL runbooks to clarify GCP snapshot best practices
   • Add a checklist for change management reviewers and approvers to the runbooks, referenced in the handbook
   • Schedule a call with Google about IO sensitivities

   Lessons

   Do not take a GCP snapshot of a running PostgreSQL leader. Technical reviewers have to review every change or addition to a change request's description before a manager or director approves it. The Datastores team should explain in detail how backups and snapshots work on the production database. No action belonging to a change should start before the change's scheduled start.

   View case
2. IL-0518 gitlab.freedesktop.org on GKE ran short of volume space and a planned 30-minute storage upgrade became a 4-hour outage

   freedesktop.org2018

   Symptoms

   Shortly after 9am UTC on 17 July 2018, the disk space of the GitLab cluster behind gitlab.freedesktop.org was critical, and some API requests were failing because of the shortage. The administrator first learned of the failure when people mentioned it on IRC.

   Persistent volume fullHigh error rateService outage

   Trigger

   Many recently created forks of Mesa, each about 2.1GB, filled the disk space of the GitLab cluster.

   Root cause

   Capacity exhaustion

   Component

   Persistent volumes and stateful setsCloud block storageHelm

   Contributing factor

   Unverified assumptionUnrepresentative stagingMonitoring blind spotChange during peak hours

   Detection

   People mentioned the failure on IRC. Monitoring and alerting on disk usage and API error rate had not been set up yet.

   Remediation

   • Dropped all inbound traffic to GitLab at the firewall except from the administrator's IP
   • Took a manual backup by streaming tar over SSH to an encrypted partition on a local machine
   • Snapshotted all GCE disks behind the PersistentVolumes
   • Copied the data from the old disk snapshots to new enlarged volumes through a temporary GCE VM
   • Re-did the copy into new empty volumes with the correct directory structure and checked repositories, users and workflows
   • Restored public access to GitLab

   Action items

   • Set up monitoring, logging and alerts
   • Find ways to communicate outages much better; this one was announced only on IRC, and the lists were not told when it dragged on
   • Use the Cloud console clients instead of the slow Cloud Console web UI
   • Add more GitLab administrators, ideally someone familiar with Kubernetes deployments
   • Move from the single Omnibus container to GitLab's cloud native deployment with one container per service
   • Move storage to resizeable volumes or to Google Cloud Storage

   Lessons

   The backup step alone used up the outage window; backups had grown along with the disk problem, and a test run before announcing the window could have measured it. The changes had been tested on a scratch cluster, but not exactly as run, and tests of steps such as disk snapshots could have been more realistic. A single GitLab administrator who was new to Kubernetes was a single point of failure. Keeping at least three copies of the data from the first destructive step meant no data was lost.

   View case
3. IL-0468 New JupyterHub student pods stuck Pending on MaxVolumeCount after a memory limit cut with node size unchanged

   UC Berkeley2017

   Symptoms

   On 20 March 2017, from some time early in the day until about 13:00, some new student servers on UC Berkeley's JupyterHub stayed in Pending forever and the students got 500 errors that kept them from logging in.

   Pods PendingHigh error rate

   Trigger

   On 18 March at 16:30 the RAM per student was reduced from 2G to 1G to save resources while the nodes kept the same size of 26G RAM, and as a result the nodes hit the Google Cloud the limit on the number of disks per node.

   Root cause

   Capacity exhaustion

   Component

   Container memory limitCloud block storageNode pool

   Contributing factor

   Unknown limit

   Detection

   New student servers reported a 500 error that kept students from logging in. At 12:44 this was judged widespread, and at 12:53 `kubectl describe pod` showed the pods Pending on MaxVolumeCount.

   Remediation

   • Copied the Google Container Engine instance template and changed it to spawn smaller n1-highmem-2 nodes
   • Switched the managed instance group to the new template
   • Grew the instance group by 30 new nodes
   • Ran populate.bash so student pods would start on time on the new nodes
   • Stopped the simple autoscaler
   • Stopped and started affected user pods from the control panel
   • Restarted the hub and proxy pods

   Action items

   • Follow the status of kubernetes/kubernetes issue 24317
   • Find and fix the cause of the 'too many redirects' error in JupyterHub
   • Delete the older, larger nodes that are no longer in use
   • Alert when any pods stay Pending for a non-negligible time

   View case

By root cause1