Cases8 cases

6 to 8 of 8

1. IL-0052 A cryptocurrency miner ran on every node of the development and staging clusters through a public, unauthenticated Weave Scope dashboard

   JW Player2019

   Symptoms

   Datadog alerted the DevOps team to a high normalised load average, first on the staging cluster (Day 0, 21:06 EST) and the next day on the development cluster (Day 1, 16:53 EST). The load was first put down to a legitimate service that kept scaling up and down in both environments, and because of the volume of identical alerts they were muted until things settled.

   Sustained high node CPUUnauthorised access or a foreign workload

   Trigger

   An automated crawler found the Weave Scope dashboard on a public load balancer that required no authentication. Weave Scope can open a shell in running containers, and its scope-agent containers ran privileged, as root and with the host's root filesystem mounted, so a command started there could run on the host.

   Root cause

   Configuration change

   Component

   Cluster dashboardCloud load balancer

   Contributing factor

   Ignored early signalUnreviewed defaultsExcessive blast radiusMonitoring blind spot

   Detection

   Datadog alerts on high normalised load average, muted at first as noise from a legitimate service; on Day 3 an engineer ran top on a node and found the gcc process at 100% CPU.

   Remediation

   • stopped Weave Scope and removed its deployment from all Kubernetes clusters
   • checked access logs for unauthorised access to the instances
   • confirmed the miner's outbound connections went only to a mining pool
   • isolated one affected node for later investigation
   • rotated out every node in each cluster so all affected hosts were destroyed and rebuilt

   Action items

   • Audit all Kubernetes and deployment YAML files and security groups for services that are unintentionally public or misconfigured
   • Replace load-average alerts with more meaningful monitors and tune them to signal a real anomaly
   • Research tools such as falcon or sysdig for behavioural monitoring and anomaly and intrusion detection
   • Evaluate Istio and Linkerd for observing and controlling end-to-end network traffic
   • Analyse the cost of scanning Docker images and containers for known vulnerabilities
   • Set aside time in architecture discussions to revisit older choices such as running containers as root
   • Share knowledge of large projects, such as Kubernetes version migrations, across the whole team

   Lessons

   Load average was not a useful signal for spotting a problem in the clusters, and muted noisy alerts nearly hid the miner. A load balancer that Kubernetes creates belongs to Kubernetes: a manual security group edit in the AWS console is reverted, so internal exposure must be declared in the Kubernetes YAML. A dashboard that can open shells, in front of privileged agents that mount the host's root filesystem, gives whoever reaches it root on every node.

   View case
2. IL-0466 Prow scheduled test job pods onto its service cluster, leaving hundreds Pending and starving Prow's own services

   Kubernetes project2019

   Symptoms

   On 8 February 2019 the Kubernetes project's Prow instance at prow.k8s.io became very slow: its controllers took about five minutes to respond to the hook, bot commands such as the cat command stopped working, and the prow.k8s.io site was hard to reach. GitHub status was normal, and the Prow service nodes were found running above 100% of CPU and memory.

   Pods PendingSustained high node CPUHigh latency

   Trigger

   At 13:20 PST an auto-generated bump PR for Prow was merged and deployed, and Prow started scheduling workloads onto the service cluster.

   Root cause

   Faulty deployment

   Component

   CI system

   Contributing factor

   Unrepresentative stagingNo resource requests or limitsLimited debug access

   Detection

   While working on a PR, a maintainer noticed the bot's cat command failing, found GitHub status normal, learned on Slack that the Prow nodes were above 100% CPU and memory, and then found 600 pending pods in the test-pods namespace.

   Remediation

   • Rolled Prow back to v20190206-b01b288 and cleaned up the pods and ProwJobs in the service cluster
   • Applied the revert by hand with kubectl apply after the deployment permission was missing
   • Sent a fix, PR 11209, so the client no longer overwrites a provided literal default client
   • Rolled Prow back to v20190125-2aca69d, from before the bug was introduced
   • Cleaned up the wrongly scheduled pods and ProwJobs again and confirmed Prow was healthy

   Action items

   • Find out why the bug did not show deterministically
   • Be more careful with sweeping Prow changes, with more end-to-end tests or a real staging Prow

   Lessons

   The behaviour was not deterministic, and the change was deployed on a Friday afternoon. Prow instances are set up differently, for example in how build clusters are defined, so another project's Prow running a version fine does not show it is safe for the Kubernetes Prow, which needs a staging environment that looks like it.

   View case
3. IL-0064 CPU-heavy pods packed onto one Google Cloud node drove it to kernel panics, and the crash moved from node to node until the site was down

   Moonlight2019

   Symptoms

   On Friday, January 18, Moonlight's API began returning occasional errors that looked like connection problems with its Redis database. The API checks Redis on every authenticated request to validate the session. At the same time the Kubernetes monitoring reported some nodes and pods as unresponsive.

   High error rateIntermittent connection failuresNode NotReadyService outageSustained high node CPU

   Trigger

   The Kubernetes scheduler put several CPU-heavy pods on the same node, including all three replicas of the website and the server-side rendering pods.

   Root cause

   Capacity exhaustion

   Component

   Pod scheduling and placementLinux kernelDatabase

   Contributing factor

   Single point of failureUnverified assumptionAutomation amplification

   Detection

   API errors that looked like Redis connectivity problems, and Kubernetes monitoring that showed nodes and pods as unresponsive. Google Cloud Support later found the link between sustained 100% CPU and kernel panics.

   Remediation

   • added anti-affinity rules to all major deployments so their pods spread across nodes

   Action items

   • Keep working with Google Cloud support to find and fix the cause of the node kernel panics

   Lessons

   Three replicas give no fault tolerance if the scheduler places them all on one node; anti-affinity rules make the spread explicit. A node failure that comes from the workload itself moves with the workload when the scheduler reschedules it.

   View case

By root cause5