IL-0460 GoCardless · 2020-10-25kubernetesgcp

Vault served an expired TLS certificate and almost every GoCardless service pod crash-looped, a 22-minute full outage

GoCardless runs HashiCorp Vault as a highly available cluster inside Kubernetes, with Google Cloud Storage as its backend, and almost all production services read their secrets from it, authenticating with their Kubernetes service account tokens. cert-manager issues Vault's short-lived Let's Encrypt certificate. When cert-manager renewed the certificate, the new data was updated in the running container, but Vault only loads a new certificate after a restart or a SIGHUP, and no automation did that. Vault kept serving the old certificate until it expired at 07:22 on 25 October 2020.

Applications get their secrets through theatre-envconsul, an in-house wrapper that runs the envconsul tool with the -once flag so that it does not restart the application whenever Vault data changes. The team had not realised that envconsul still renews its Vault leases periodically in this mode, and exits with an error, killing the application process, when a renewal fails. Vault retries had also been disabled in envconsul to work around other problems. As leases came up for renewal, the healthy pods died one by one and went into crash loops, and by 07:31 no backend could serve requests.

The on-call engineer was paged at 07:32. Believing the certificate itself had expired, the responders forced cert-manager to recreate it at 07:42, a step that was not needed and took minutes because of DNS validation. At 07:48 the Vault cluster was restarted by hand to load the new certificate, Kubernetes brought up new containers, and some pods of critical workloads were deleted to skip the 5-minute exponential backoff they had reached. Backends were serving again at 07:52.

Severity: –
Scope: Full outage of the GoCardless API, dashboard, payment pages and operations through partner integrations for 22 minutes
Users affected: –
Region: –
Time to detect: 10m
Time to resolve: 32m

Symptoms

On Sunday 25 October 2020 the GoCardless API, dashboard, payment pages and partner integrations were all unavailable for 22 minutes, from 07:32 to 07:54 GMT. Automated monitoring paged the on-call engineer at 07:32, a minute after every request to the API had started failing because no healthy backend was left.

The TLS certificate that the Vault secrets server presented on its API expired at 07:22. From 07:25 application containers began crash-looping: envconsul, the tool that injects secrets from Vault, exited when it could not renew its Vault leases, took the healthy application process with it, and could not connect to Vault again on restart. Pods serving production traffic died one by one as their leases came up for renewal.

The certificate had in fact been renewed by cert-manager and the new data was mounted into the Vault container, but Vault kept serving the old certificate. The responders believed the certificate itself had expired and at 07:42 forced cert-manager to recreate it, which took several minutes because issuance uses DNS validation.

Symptom Service outage Container crash loop TLS certificate errors Credentials unavailable

Root cause

Root cause Certificate or credential expiry

Component cert-manager Secrets store

Contributing factor Single point of failure Monitoring blind spot Unverified assumption Harmful manual action

Trigger: At 07:22 the TLS certificate Vault was still serving expired. cert-manager had already renewed it, but Vault does not load a new certificate without a restart or a SIGHUP.

Detection

Automated monitoring detected the failing requests and paged the on-call engineer at 07:32. Certificate expiry monitoring existed for the Layer-7 load balancers but did not cover the TCP load balancer that Vault uses.

Remediation

Forced cert-manager to recreate the Vault certificate
Restarted the Vault cluster by hand so it loaded the new certificate
Deleted some pods of critical workloads to skip their 5-minute exponential backoff

Action items

Check the certificate of the TCP load balancer in front of Vault with the Blackbox exporter, as for the Layer-7 load balancers
Deploy the Reloader controller to roll Vault when its certificate data changes
Change theatre-envconsul so envconsul no longer manages the child process and only injects secrets at container startup

Lessons

A system that almost every service depends on, such as the secrets store, can take everything down at once, and failure scenarios like Vault becoming unreachable had not been tested enough. Assuming the certificate had expired, when it had been renewed but not reloaded, added a slow and unnecessary recreation step to the recovery.

Sources

https://gocardless.com/blog/incident-review-service-outage-on-25-october-2020/ · blog · archived copy

Related cases

–

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, cert-manager, Secrets store among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0460
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0460/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.