Incident Library
Applied filters Root causeCertificate or credential expiryDomainKubernetesClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases8 cases

8 to 8 of 8

  1. IL-0460 Vault served an expired TLS certificate and almost every GoCardless service pod crash-looped, a 22-minute full outage
    GoCardless2020
    Symptoms

    On Sunday 25 October 2020 the GoCardless API, dashboard, payment pages and partner integrations were all unavailable for 22 minutes, from 07:32 to 07:54 GMT. Automated monitoring paged the on-call engineer at 07:32, a minute after every request to the API had started failing because no healthy backend was left.

    Service outageContainer crash loopTLS certificate errorsCredentials unavailable
    Trigger

    At 07:22 the TLS certificate Vault was still serving expired. cert-manager had already renewed it, but Vault does not load a new certificate without a restart or a SIGHUP.

    Root cause
    Certificate or credential expiry
    Component
    cert-managerSecrets store
    Contributing factor
    Single point of failureMonitoring blind spotUnverified assumptionHarmful manual action
    Detection

    Automated monitoring detected the failing requests and paged the on-call engineer at 07:32. Certificate expiry monitoring existed for the Layer-7 load balancers but did not cover the TCP load balancer that Vault uses.

    Remediation
    • Forced cert-manager to recreate the Vault certificate
    • Restarted the Vault cluster by hand so it loaded the new certificate
    • Deleted some pods of critical workloads to skip their 5-minute exponential backoff
    Action items
    • Check the certificate of the TCP load balancer in front of Vault with the Blackbox exporter, as for the Layer-7 load balancers
    • Deploy the Reloader controller to roll Vault when its certificate data changes
    • Change theatre-envconsul so envconsul no longer manages the child process and only injects secrets at container startup
    Lessons

    A system that almost every service depends on, such as the secrets store, can take everything down at once, and failure scenarios like Vault becoming unreachable had not been tested enough. Assuming the certificate had expired, when it had been renewed but not reloaded, added a slow and unnecessary recreation step to the recovery.

    View case
Check your cluster for this

These 8 cases involve Kubernetes API server and Node pool. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Certificate or credential expiry8 cases

Component
Kubernetes API server3Node pool3kubelet2Admission webhook1Calico1cert-manager1Cluster autoscaler1Cluster cloud identity1CNI plugin1etcd1GKE control plane1Managed Kubernetes control plane and provider API1Persistent volumes and stateful sets1Policy engine1RBAC and service accounts1Secrets store1
Contributing factor
Monitoring blind spot3Unreviewed defaults2Single point of failure2Unverified assumption1Change with a delayed effect1Ignored early signal1Unknown limit1Limited debug access1Harmful manual action1No alert on limit1Deferred upgrades1
Organisation
Linode2ckck8031dbhang1DFDS1GoCardless1Jenkins project1Webhook Relay1
  1. IL-0158 An expired credential caused authentication failures that stopped LKE and LKE-E from deploying clusters and nodes in all regions, which also blocked node recycling and cluster autoscaling
    Linode2026
  2. IL-0183 kubeadm's one-year leaf certificates expired, so kube-apiserver could not reach etcd, restarted in a loop and left the cluster unreachable
    ckck8032026
  3. IL-0156 Revoking old certificates after a rotation left a certificate key mismatch between API endpoints and the cluster controller, and LKE cluster management and autoscaling in Atlanta returned 5xx errors
    Linode2025
  4. IL-0107 Kyverno stuck in a crash loop behind a webhook with failurePolicy fail blocked every change to Kubernetes Service objects for 88 minutes
    DFDS2022
  5. IL-0113 The managed GKE control plane of Webhook Relay's main cluster failed because its API server certificates had never been rotated, and a manual credential rotation reported success without taking effect
    Webhook Relay2022
  6. IL-0521 AKS pods could not mount volumes and new nodes got no CNI network interface after the cluster service principal expired
    Jenkins project2021
  7. IL-0184 JupyterHub's KubeSpawner could not create containers and kubectl was refused because the API server and other internal certificates of a kubeadm cluster had expired
    dbhang2021
  8. IL-0460 Vault served an expired TLS certificate and almost every GoCardless service pod crash-looped, a 22-minute full outage
    GoCardless2020