Cases43 cases

41 to 43 of 43

1. IL-0051 A third-party log shipping client leaked file descriptors with every new container until nodes failed with too many open files

   thredUP2018

   Symptoms

   While thredUP's move to Kubernetes seemed to be going well, Datadog showed the number of open files on the nodes climbing steadily, with no obvious reason.

   Service outageFile descriptor exhaustion

   Trigger

   Containers starting on a node, which in Kubernetes happens constantly with every deploy and every pod that moves between nodes.

   Root cause

   Software defect

   Component

   Telemetry agent

   Contributing factor

   Unverified assumption

   Detection

   Datadog showed the number of open files on the nodes rising.

   Remediation

   • killed the affected node, brought up a new one and redeployed

   Action items

   • Move away from the log aggregation vendor

   Lessons

   Libraries and agents that worked with the EC2 deployment lifecycle may not cope with Kubernetes, where containers start, stop and move between nodes constantly. Each one needs to be checked against the new lifecycle instead of being trusted because it worked before.

   View case
2. IL-0518 gitlab.freedesktop.org on GKE ran short of volume space and a planned 30-minute storage upgrade became a 4-hour outage

   freedesktop.org2018

   Symptoms

   Shortly after 9am UTC on 17 July 2018, the disk space of the GitLab cluster behind gitlab.freedesktop.org was critical, and some API requests were failing because of the shortage. The administrator first learned of the failure when people mentioned it on IRC.

   Persistent volume fullHigh error rateService outage

   Trigger

   Many recently created forks of Mesa, each about 2.1GB, filled the disk space of the GitLab cluster.

   Root cause

   Capacity exhaustion

   Component

   Persistent volumes and stateful setsCloud block storageHelm

   Contributing factor

   Unverified assumptionUnrepresentative stagingMonitoring blind spotChange during peak hours

   Detection

   People mentioned the failure on IRC. Monitoring and alerting on disk usage and API error rate had not been set up yet.

   Remediation

   • Dropped all inbound traffic to GitLab at the firewall except from the administrator's IP
   • Took a manual backup by streaming tar over SSH to an encrypted partition on a local machine
   • Snapshotted all GCE disks behind the PersistentVolumes
   • Copied the data from the old disk snapshots to new enlarged volumes through a temporary GCE VM
   • Re-did the copy into new empty volumes with the correct directory structure and checked repositories, users and workflows
   • Restored public access to GitLab

   Action items

   • Set up monitoring, logging and alerts
   • Find ways to communicate outages much better; this one was announced only on IRC, and the lists were not told when it dragged on
   • Use the Cloud console clients instead of the slow Cloud Console web UI
   • Add more GitLab administrators, ideally someone familiar with Kubernetes deployments
   • Move from the single Omnibus container to GitLab's cloud native deployment with one container per service
   • Move storage to resizeable volumes or to Google Cloud Storage

   Lessons

   The backup step alone used up the outage window; backups had grown along with the disk problem, and a test run before announcing the window could have measured it. The changes had been tested on a scratch cluster, but not exactly as run, and tests of steps such as disk snapshots could have been more realistic. A single GitLab administrator who was new to Kubernetes was a single point of failure. Keeping at least three copies of the data from the first destructive step meant no data was lost.

   View case
3. IL-0036 A reindex Job that kept failing exhausted one node after another until the whole cluster was down

   Universe2017

   Symptoms

   Shortly after a search reindex job was started, problems with the Kubernetes cluster hit Universe's reporting and search APIs and the Kubernetes cluster UI itself. The status page opened an investigation into reports of search and reporting being unavailable at 14:37 EDT on 2017-07-21.

   Service outageNode NotReadyNode recreation loop

   Trigger

   A change to the search index definition, which removed some fields from full-text indexing, was shipped, and a reindex job was started as a Kubernetes Job to apply it.

   Root cause

   Capacity exhaustion

   Component

   Jobs and CronJobsPod scheduling and placement

   Contributing factor

   Unverified assumptionAutomation amplificationExcessive blast radius

   Detection

   The team saw problems with the Kubernetes cluster shortly after starting the job, and the status page cites reports of search and reporting being unavailable.

   Remediation

   • removed the reindex Job from the scheduler
   • let the nodes auto-heal back to healthy

   Lessons

   restartPolicy: Never on the pod template does not stop a Kubernetes Job from retrying: a Job replaces failed pods and may run more than once, so a job that can exhaust a node will keep moving to new nodes.

   View case

By root cause10