Cases2 cases

1 to 2 of 2

1. IL-0521 AKS pods could not mount volumes and new nodes got no CNI network interface after the cluster service principal expired

   Jenkins project2021

   Symptoms

   From 18:02 UTC on 2 June 2021, the Jenkins infrastructure AKS cluster logged errors about pods that were unable to mount volumes and about scaled-up nodes that were unable to obtain a network virtual interface from the CNI.

   Credentials unavailablePods stuck in ContainerCreating

   Trigger

   The Azure Service Principal credential of the AKS cluster expired at 17:53 UTC on 2 June 2021.

   Root cause

   Certificate or credential expiry

   Component

   Cluster cloud identityPersistent volumes and stateful setsCNI pluginNode pool

   Contributing factor

   Monitoring blind spot

   Detection

   The error behind the failures was found in the AKS UI console while the team investigated the root cause of the LTS 2.289.1 release problem.

   Remediation

   • Renewed the AKS cluster's Azure Service Principal credential, which made AKS restart all nodes automatically

   Action items

   • Add an automatic PagerDuty check that alerts a week before the service principal expires, as is done for certificates
   • Keep a team-level calendar for manual operations such as resetting a credential
   • Consider managed identities, which renew automatically, instead of the service principal

   View case
2. IL-0460 Vault served an expired TLS certificate and almost every GoCardless service pod crash-looped, a 22-minute full outage

   GoCardless2020

   Symptoms

   On Sunday 25 October 2020 the GoCardless API, dashboard, payment pages and partner integrations were all unavailable for 22 minutes, from 07:32 to 07:54 GMT. Automated monitoring paged the on-call engineer at 07:32, a minute after every request to the API had started failing because no healthy backend was left.

   Service outageContainer crash loopTLS certificate errorsCredentials unavailable

   Trigger

   At 07:22 the TLS certificate Vault was still serving expired. cert-manager had already renewed it, but Vault does not load a new certificate without a restart or a SIGHUP.

   Root cause

   Certificate or credential expiry

   Component

   cert-managerSecrets store

   Contributing factor

   Single point of failureMonitoring blind spotUnverified assumptionHarmful manual action

   Detection

   Automated monitoring detected the failing requests and paged the on-call engineer at 07:32. Certificate expiry monitoring existed for the Layer-7 load balancers but did not cover the TCP load balancer that Vault uses.

   Remediation

   • Forced cert-manager to recreate the Vault certificate
   • Restarted the Vault cluster by hand so it loaded the new certificate
   • Deleted some pods of critical workloads to skip their 5-minute exponential backoff

   Action items

   • Check the certificate of the TCP load balancer in front of Vault with the Blackbox exporter, as for the Layer-7 load balancers
   • Deploy the Reloader controller to roll Vault when its certificate data changes
   • Change theatre-envconsul so envconsul no longer manages the child process and only injects secrets at container startup

   Lessons

   A system that almost every service depends on, such as the secrets store, can take everything down at once, and failure scenarios like Vault becoming unreachable had not been tested enough. Assuming the certificate had expired, when it had been renewed but not reloaded, added a slow and unnecessary recreation step to the recovery.

   View case

By root cause1