Cases5 cases

1 to 5 of 5

1. IL-0141 gke-metadata-server versions 0.4.272 to 0.4.280 flooded Cloud Logging with "Unable to sync sandbox" debug logs on GKE clusters with Workload Identity, using up logging quota and adding billable usage

   Google Cloud2023

   Symptoms

   Google Kubernetes Engine (GKE) customers with Workload Identity enabled saw a high logging rate. Debug log lines containing the string "Unable to sync sandbox" were ingested into Cloud Logging, used up the Cloud Logging ingestion quota and, beyond the free monthly allotment, produced extra billable usage. Workload Identity is opt-in on GKE Standard and always enabled on GKE Autopilot.

   Log floodHigh latency

   Trigger

   GKE clusters ran gke-metadata-server versions 0.4.272 to 0.4.280, which contained an incorrect configuration that wrote "Unable to sync sandbox" debug logs at a high rate.

   Root cause

   Software defect

   Component

   Credential broker for pods

   Contributing factor

   –

   Detection

   The first status update was posted at 13:32 US/Pacific on 2023-11-02, and the updates told customers how to check a cluster from the version annotation on the gke-metadata-server DaemonSet.

   Remediation

   • rolled out a fix across the GKE fleet so the excess logs were no longer ingested into Cloud Logging
   • offered GKE Rapid channel customers a control plane upgrade to 1.28.2-gke.1157000 or above, or 1.27.7-gke.1038000 or above, as a workaround

   View case
2. IL-0089 A test cluster deletion script run with the kube context pointing at the live-1 production cluster deleted its ingress controller, monitoring, logging, cert-manager, kiam and external-dns

   UK Ministry of Justice2020

   Symptoms

   At 18:27 on 21 September 2020 the Cloud Platform team detected that some services running on the live-1 production cluster had gone down, and some users could not reach them. The incident was declared at 18:40.

   Service outageResources deletedMonitoring outageStale endpoint routing

   Trigger

   A script for deleting a test cluster was triggered while the kube context was pointing at the live-1 production cluster instead of the test cluster.

   Root cause

   Operator error

   Component

   Ingress controllerCredential broker for podsexternal-dnsCloud load balancerPrometheus

   Contributing factor

   No safeguard on destructive change

   Detection

   The first detection was at 18:27, and the incident was declared 13 minutes later, at 18:40.

   Remediation

   • restored the default ingress controller together with external-dns and kiam, which it depends on
   • corrected the owner field in the Formbuilder Route 53 TXT records so external-dns pointed their A records at the new NLB
   • restored the remaining components with a targeted apply
   • ran the apply pipeline to restore all certificates, ServiceMonitors and Prometheus rules from the environment repository

   View case
3. IL-0189 A JVM service moved from EC2 to Kubernetes answered up to ten times slower because KIAM's 15-minute credentials made the AWS Java SDK fetch new credentials on every request

   Adevinta2019

   Symptoms

   After a Spring-based JVM microservice was deployed to Kubernetes and part of production traffic was routed to it, request latency there was up to 10 times higher than on EC2. Behind the NGINX ingress controller the backend took 100 to 200 ms to answer, where the EC2 instances took about 20 ms.

   High latency

   Trigger

   Moving the service onto the Kubernetes platform, where KIAM served IAM credentials to pods with its default 15-minute lifetime.

   Root cause

   Configuration change

   Component

   Credential broker for pods

   Contributing factor

   Unreviewed defaults

   Detection

   After part of production traffic was routed to the Kubernetes deployment, its request latency was seen to be up to 10 times higher than on EC2.

   Remediation

   • reconfigured KIAM to request IAM credentials with a longer expiration period

   Lessons

   KIAM's short credential lifetime for pods and the AWS Java SDK's eager refresh of any credential within 15 minutes of expiry are each reasonable, but together they force a metadata round trip on every request. When latency rises after a move to Kubernetes, the cause is often the interaction of components that never ran together before rather than Kubernetes itself. Here a packet capture from inside the container found what latency metrics and DNS tests had not.

   View case
4. IL-0002 CI builder pods could not get AWS IAM credentials because kubelet fell minutes behind its API rate limit

   Zalando2019

   Symptoms

   Builds on Zalando's in-house Continuous Delivery Platform (CDP), which runs CI/CD on Kubernetes, timed out and failed for almost an hour on 11 June 2019. The Kubernetes builder pods could not get AWS IAM credentials.

   High error rateCredentials unavailableControl plane lag

   Trigger

   An unlucky rescaling of the CDP Kubernetes cluster left only one node available for builder pods. Builder pods were created and deleted on that node so quickly that kubelet fell behind.

   Root cause

   Capacity exhaustion

   Component

   kubeletCredential broker for podsKubernetes API serverCI system

   Contributing factor

   Unreviewed defaultsSingle point of failure

   Remediation

   • Scaled the CDP cluster up by hand so builder pods spread over more than one node

   Action items

   • Raise the kubelet QPS limit from the default 5 to 50 in the global cluster configuration

   View case
5. IL-0073 Intermittent Kafka connectivity overloaded Docker daemons, set off a node rescheduling cascade and poisoned the Consul gossip mesh

   Target2019sev1

   Symptoms

   An upgrade of the network subsystem under Target's datacenter OpenStack, meant to cause only a brief disruption, ran into problems and left network connectivity disrupted for several hours. Kafka, which runs on that OpenStack, became intermittently available.

   Node NotReadyLoad imbalanceHigh latencyCredentials unavailable

   Trigger

   A network subsystem upgrade in the datacenter OpenStack infrastructure disrupted connectivity for several hours and made Kafka intermittently available.

   Root cause

   Dependency failure

   Component

   Container runtimeService discoveryTelemetry agentCredential broker for pods

   Contributing factor

   Excessive blast radiusAutomation amplificationSingle point of failureUnpatched known bug

   Remediation

   • enabled gossip encryption on Consul so messages from the poisoned mesh were rejected
   • stabilized the deployment engine and redeployed the development environment with encryption, terminating instances without it
   • upgraded Consul in place to 1.2.3 and then to 1.3.0

   Action items

   • Reduce how much of an outage a single Docker daemon can cause, together with running smaller and more numerous clusters

   Lessons

   Smaller and more numerous clusters limited the damage: the smaller development clusters and production were not hit the same way as the one large cluster, and dev and prod on one cluster would have been far worse. A Docker daemon shared by many workloads is a brittle failure point. Per-workload logging and metrics sidecars were judged the right design; a shared cluster-level agent would almost certainly have made the problem worse and harder to fix.

   View case

By root cause5