Cases2 cases

1 to 2 of 2

1. IL-0034 A crashing stats agent sidecar marked every pod unready and the service mesh pulled all services out of traffic

   Airbnb2019

   Symptoms

   An operation to upgrade the stats agent in Kubernetes was announced in advance, with a warning that metrics might be unavailable to services for a few minutes. A few minutes later a reply came in saying that all of the sender's Kubernetes services seemed to be down.

   Probe failuresService outage

   Trigger

   A scheduled upgrade of the stats agent sidecar, during which the agent container crashed.

   Root cause

   Faulty deployment

   Component

   Telemetry agentService discovery

   Contributing factor

   Unverified assumptionExcessive blast radius

   Detection

   A reply to the operation announcement, a few minutes later, said that all of the sender's Kubernetes services appeared to be down.

   Action items

   • Mark logging, tracing and stats agent containers as non-critical and leave them out of the service mesh's pod readiness check

   Lessons

   Every container in a pod affects the pod's readiness, so each sidecar needs a deliberate decision about how its health should count. A sidecar that crashes is not ready even when it has no probes, and any logic that requires all containers to be ready will take the pod out of traffic.

   View case
2. IL-0073 Intermittent Kafka connectivity overloaded Docker daemons, set off a node rescheduling cascade and poisoned the Consul gossip mesh

   Target2019sev1

   Symptoms

   An upgrade of the network subsystem under Target's datacenter OpenStack, meant to cause only a brief disruption, ran into problems and left network connectivity disrupted for several hours. Kafka, which runs on that OpenStack, became intermittently available.

   Node NotReadyLoad imbalanceHigh latencyCredentials unavailable

   Trigger

   A network subsystem upgrade in the datacenter OpenStack infrastructure disrupted connectivity for several hours and made Kafka intermittently available.

   Root cause

   Dependency failure

   Component

   Container runtimeService discoveryTelemetry agentCredential broker for pods

   Contributing factor

   Excessive blast radiusAutomation amplificationSingle point of failureUnpatched known bug

   Remediation

   • enabled gossip encryption on Consul so messages from the poisoned mesh were rejected
   • stabilized the deployment engine and redeployed the development environment with encryption, terminating instances without it
   • upgraded Consul in place to 1.2.3 and then to 1.3.0

   Action items

   • Reduce how much of an outage a single Docker daemon can cause, together with running smaller and more numerous clusters

   Lessons

   Smaller and more numerous clusters limited the damage: the smaller development clusters and production were not hit the same way as the one large cluster, and dev and prod on one cluster would have been far worse. A Docker daemon shared by many workloads is a brittle failure point. Per-workload logging and metrics sidecars were judged the right design; a shared cluster-level agent would almost certainly have made the problem worse and harder to fix.

   View case

By root cause2