Incident Library
Applied filters SymptomTLS certificate errorsFactorUnverified assumptionDomainKubernetesClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0460 Vault served an expired TLS certificate and almost every GoCardless service pod crash-looped, a 22-minute full outage
    GoCardless2020
    Symptoms

    On Sunday 25 October 2020 the GoCardless API, dashboard, payment pages and partner integrations were all unavailable for 22 minutes, from 07:32 to 07:54 GMT. Automated monitoring paged the on-call engineer at 07:32, a minute after every request to the API had started failing because no healthy backend was left.

    Service outageContainer crash loopTLS certificate errorsCredentials unavailable
    Trigger

    At 07:22 the TLS certificate Vault was still serving expired. cert-manager had already renewed it, but Vault does not load a new certificate without a restart or a SIGHUP.

    Root cause
    Certificate or credential expiry
    Component
    cert-managerSecrets store
    Contributing factor
    Single point of failureMonitoring blind spotUnverified assumptionHarmful manual action
    Detection

    Automated monitoring detected the failing requests and paged the on-call engineer at 07:32. Certificate expiry monitoring existed for the Layer-7 load balancers but did not cover the TCP load balancer that Vault uses.

    Remediation
    • Forced cert-manager to recreate the Vault certificate
    • Restarted the Vault cluster by hand so it loaded the new certificate
    • Deleted some pods of critical workloads to skip their 5-minute exponential backoff
    Action items
    • Check the certificate of the TCP load balancer in front of Vault with the Blackbox exporter, as for the Layer-7 load balancers
    • Deploy the Reloader controller to roll Vault when its certificate data changes
    • Change theatre-envconsul so envconsul no longer manages the child process and only injects secrets at container startup
    Lessons

    A system that almost every service depends on, such as the secrets store, can take everything down at once, and failure scenarios like Vault becoming unreachable had not been tested enough. Assuming the certificate had expired, when it had been renewed but not reloaded, added a slow and unnecessary recreation step to the recovery.

    View case
Check your cluster for this

This case involves cert-manager and Secrets store. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Certificate or credential expiry1 case

Component
cert-managerSecrets store
Contributing factor
Unverified assumptionHarmful manual actionMonitoring blind spotSingle point of failure
Organisation
GoCardless
  1. IL-0460 Vault served an expired TLS certificate and almost every GoCardless service pod crash-looped, a 22-minute full outage
    GoCardless2020