Cases2 cases

1 to 2 of 2

1. IL-0206 Consul agents in GitLab.com's GKE pods were repeatedly OOM-killed, so the Rails pods lost database replica discovery and sent every query to the primary

   GitLab2022

   Symptoms

   On 2022-04-25 the Sidekiq Apdex (the share of background jobs that finish within a target time) dropped, and an engineer declared an incident in Slack at 15:13 UTC. Users saw intermittent slowness across GitLab.com, and asynchronous work such as pipeline creation and import and export took unusually long.

   High latencyDatabase overloadOOMKilledHigh error rate

   Trigger

   The Consul agent in GitLab.com's GKE pods had memory demand spikes that did not fit in the pods' 500MB memory limit, and the agent was killed again and again.

   Root cause

   Capacity exhaustion

   Component

   Container memory limitService discoveryApplication database

   Contributing factor

   –

   Detection

   An engineer declared the incident in Slack at 15:13 UTC after the Sidekiq Apdex dropped; the second event was found when monitoring detected an elevated error rate at 01:21 UTC on 2022-04-26.

   Remediation

   • planned at 16:13 to raise the per-user connection limit for the gitlab database user and the pgbouncer backend pools for the main database; changes were applied at 17:26 and confirmed at 17:47
   • raised the pod memory limits from 500MB to 1GB so the Consul agents survive memory spikes

   Action items

   • Monitor the Consul agents on GKE pods to see whether the 1GB memory limit keeps them healthy
   • Investigate the Consul memory demand spikes and what can be done beyond raising the hard memory limit

   Lessons

   When the local Consul agent that Rails uses for database service discovery keeps dying, Rails does not fail; it sends every query to the last known primary. What engineers saw was database saturation and slow background jobs, which pointed the investigation first at a recent pgbouncer pool change. The memory limit of the Consul agent in every pod therefore decided the load on the primary database.

   View case
2. IL-0073 Intermittent Kafka connectivity overloaded Docker daemons, set off a node rescheduling cascade and poisoned the Consul gossip mesh

   Target2019sev1

   Symptoms

   An upgrade of the network subsystem under Target's datacenter OpenStack, meant to cause only a brief disruption, ran into problems and left network connectivity disrupted for several hours. Kafka, which runs on that OpenStack, became intermittently available.

   Node NotReadyLoad imbalanceHigh latencyCredentials unavailable

   Trigger

   A network subsystem upgrade in the datacenter OpenStack infrastructure disrupted connectivity for several hours and made Kafka intermittently available.

   Root cause

   Dependency failure

   Component

   Container runtimeService discoveryTelemetry agentCredential broker for pods

   Contributing factor

   Excessive blast radiusAutomation amplificationSingle point of failureUnpatched known bug

   Remediation

   • enabled gossip encryption on Consul so messages from the poisoned mesh were rejected
   • stabilized the deployment engine and redeployed the development environment with encryption, terminating instances without it
   • upgraded Consul in place to 1.2.3 and then to 1.3.0

   Action items

   • Reduce how much of an outage a single Docker daemon can cause, together with running smaller and more numerous clusters

   Lessons

   Smaller and more numerous clusters limited the damage: the smaller development clusters and production were not hit the same way as the one large cluster, and dev and prod on one cluster would have been far worse. A Docker daemon shared by many workloads is a brittle failure point. Per-workload logging and metrics sidecars were judged the right design; a shared cluster-level agent would almost certainly have made the problem worse and harder to fix.

   View case

By root cause2