Cases7 cases

4 to 7 of 7

1. IL-0521 AKS pods could not mount volumes and new nodes got no CNI network interface after the cluster service principal expired

   Jenkins project2021

   Symptoms

   From 18:02 UTC on 2 June 2021, the Jenkins infrastructure AKS cluster logged errors about pods that were unable to mount volumes and about scaled-up nodes that were unable to obtain a network virtual interface from the CNI.

   Credentials unavailablePods stuck in ContainerCreating

   Trigger

   The Azure Service Principal credential of the AKS cluster expired at 17:53 UTC on 2 June 2021.

   Root cause

   Certificate or credential expiry

   Component

   Cluster cloud identityPersistent volumes and stateful setsCNI pluginNode pool

   Contributing factor

   Monitoring blind spot

   Detection

   The error behind the failures was found in the AKS UI console while the team investigated the root cause of the LTS 2.289.1 release problem.

   Remediation

   • Renewed the AKS cluster's Azure Service Principal credential, which made AKS restart all nodes automatically

   Action items

   • Add an automatic PagerDuty check that alerts a week before the service principal expires, as is done for certificates
   • Keep a team-level calendar for manual operations such as resetting a credential
   • Consider managed identities, which renew automatically, instead of the service principal

   View case
2. IL-0460 Vault served an expired TLS certificate and almost every GoCardless service pod crash-looped, a 22-minute full outage

   GoCardless2020

   Symptoms

   On Sunday 25 October 2020 the GoCardless API, dashboard, payment pages and partner integrations were all unavailable for 22 minutes, from 07:32 to 07:54 GMT. Automated monitoring paged the on-call engineer at 07:32, a minute after every request to the API had started failing because no healthy backend was left.

   Service outageContainer crash loopTLS certificate errorsCredentials unavailable

   Trigger

   At 07:22 the TLS certificate Vault was still serving expired. cert-manager had already renewed it, but Vault does not load a new certificate without a restart or a SIGHUP.

   Root cause

   Certificate or credential expiry

   Component

   cert-managerSecrets store

   Contributing factor

   Single point of failureMonitoring blind spotUnverified assumptionHarmful manual action

   Detection

   Automated monitoring detected the failing requests and paged the on-call engineer at 07:32. Certificate expiry monitoring existed for the Layer-7 load balancers but did not cover the TCP load balancer that Vault uses.

   Remediation

   • Forced cert-manager to recreate the Vault certificate
   • Restarted the Vault cluster by hand so it loaded the new certificate
   • Deleted some pods of critical workloads to skip their 5-minute exponential backoff

   Action items

   • Check the certificate of the TCP load balancer in front of Vault with the Blackbox exporter, as for the Layer-7 load balancers
   • Deploy the Reloader controller to roll Vault when its certificate data changes
   • Change theatre-envconsul so envconsul no longer manages the child process and only injects secrets at container startup

   Lessons

   A system that almost every service depends on, such as the secrets store, can take everything down at once, and failure scenarios like Vault becoming unreachable had not been tested enough. Assuming the certificate had expired, when it had been renewed but not reloaded, added a slow and unnecessary recreation step to the recovery.

   View case
3. IL-0002 CI builder pods could not get AWS IAM credentials because kubelet fell minutes behind its API rate limit

   Zalando2019

   Symptoms

   Builds on Zalando's in-house Continuous Delivery Platform (CDP), which runs CI/CD on Kubernetes, timed out and failed for almost an hour on 11 June 2019. The Kubernetes builder pods could not get AWS IAM credentials.

   High error rateCredentials unavailableControl plane lag

   Trigger

   An unlucky rescaling of the CDP Kubernetes cluster left only one node available for builder pods. Builder pods were created and deleted on that node so quickly that kubelet fell behind.

   Root cause

   Capacity exhaustion

   Component

   kubeletCredential broker for podsKubernetes API serverCI system

   Contributing factor

   Unreviewed defaultsSingle point of failure

   Remediation

   • Scaled the CDP cluster up by hand so builder pods spread over more than one node

   Action items

   • Raise the kubelet QPS limit from the default 5 to 50 in the global cluster configuration

   View case
4. IL-0073 Intermittent Kafka connectivity overloaded Docker daemons, set off a node rescheduling cascade and poisoned the Consul gossip mesh

   Target2019sev1

   Symptoms

   An upgrade of the network subsystem under Target's datacenter OpenStack, meant to cause only a brief disruption, ran into problems and left network connectivity disrupted for several hours. Kafka, which runs on that OpenStack, became intermittently available.

   Node NotReadyLoad imbalanceHigh latencyCredentials unavailable

   Trigger

   A network subsystem upgrade in the datacenter OpenStack infrastructure disrupted connectivity for several hours and made Kafka intermittently available.

   Root cause

   Dependency failure

   Component

   Container runtimeService discoveryTelemetry agentCredential broker for pods

   Contributing factor

   Excessive blast radiusAutomation amplificationSingle point of failureUnpatched known bug

   Remediation

   • enabled gossip encryption on Consul so messages from the poisoned mesh were rejected
   • stabilized the deployment engine and redeployed the development environment with encryption, terminating instances without it
   • upgraded Consul in place to 1.2.3 and then to 1.3.0

   Action items

   • Reduce how much of an outage a single Docker daemon can cause, together with running smaller and more numerous clusters

   Lessons

   Smaller and more numerous clusters limited the damage: the smaller development clusters and production were not hit the same way as the one large cluster, and dev and prod on one cluster would have been far worse. A Docker daemon shared by many workloads is a brittle failure point. Per-workload logging and metrics sidecars were judged the right design; a shared cluster-level agent would almost certainly have made the problem worse and harder to fix.

   View case

By root cause4