Cases3 cases

1 to 3 of 3

1. IL-0460 Vault served an expired TLS certificate and almost every GoCardless service pod crash-looped, a 22-minute full outage

   GoCardless2020

   Symptoms

   On Sunday 25 October 2020 the GoCardless API, dashboard, payment pages and partner integrations were all unavailable for 22 minutes, from 07:32 to 07:54 GMT. Automated monitoring paged the on-call engineer at 07:32, a minute after every request to the API had started failing because no healthy backend was left.

   Service outageContainer crash loopTLS certificate errorsCredentials unavailable

   Trigger

   At 07:22 the TLS certificate Vault was still serving expired. cert-manager had already renewed it, but Vault does not load a new certificate without a restart or a SIGHUP.

   Root cause

   Certificate or credential expiry

   Component

   cert-managerSecrets store

   Contributing factor

   Single point of failureMonitoring blind spotUnverified assumptionHarmful manual action

   Detection

   Automated monitoring detected the failing requests and paged the on-call engineer at 07:32. Certificate expiry monitoring existed for the Layer-7 load balancers but did not cover the TCP load balancer that Vault uses.

   Remediation

   • Forced cert-manager to recreate the Vault certificate
   • Restarted the Vault cluster by hand so it loaded the new certificate
   • Deleted some pods of critical workloads to skip their 5-minute exponential backoff

   Action items

   • Check the certificate of the TCP load balancer in front of Vault with the Blackbox exporter, as for the Layer-7 load balancers
   • Deploy the Reloader controller to roll Vault when its certificate data changes
   • Change theatre-envconsul so envconsul no longer manages the child process and only injects secrets at container startup

   Lessons

   A system that almost every service depends on, such as the secrets store, can take everything down at once, and failure scenarios like Vault becoming unreachable had not been tested enough. Assuming the certificate had expired, when it had been renewed but not reloaded, added a slow and unnecessary recreation step to the recovery.

   View case
2. IL-0219 GoCardless's API and Dashboard failed every request for 1 hour 50 minutes when Pacemaker did not promote a replica

   GoCardless2017

   Symptoms

   At 15:09 BST on 10 October 2017, GoCardless's monitoring detected a total outage of its API and Dashboard: every request failed and returned an error. By 15:11 engineers saw evidence of a disk array failure on the primary Postgres node, but the Pacemaker cluster had not failed over to the synchronous replica as it should have.

   Service outageFailover failure

   Trigger

   The RAID controller of the primary Postgres node lost 3 disks of its array at once, so all reads and writes on the primary failed.

   Root cause

   Hardware failure

   Component

   DatabaseDatabase failover manager

   Contributing factor

   Overlapping failuresUnverified assumption

   Detection

   GoCardless's monitoring detected the total outage of the API and Dashboard at 15:09 BST, and engineers began investigating.

   Remediation

   • Powered off the broken primary Postgres node
   • Cleared the Pacemaker error counts with crm resource cleanup, which had no effect
   • Edited recovery.conf on the synchronous replica under maintenance mode several times, which Pacemaker reverted
   • Put the cluster into maintenance mode, configured the synchronous replica as primary and started Postgres by hand
   • Reconfigured the backend applications to connect to the new primary's actual IP address
   • Bootstrapped a third node as an asynchronous replica
   • Added a VIP managed by hand by the infrastructure team for manual failover
   • Verified data integrity and found no corruption
   • Migrated to a new Pacemaker-managed database cluster in a maintenance window and decommissioned the old one

   Action items

   • Express the Backup VIP rule as an INF preference for a replica instead of a -INF preference against the primary
   • Replace the VIPs with proxies on the application servers that route traffic by the state of the cluster
   • Run harsher fault injection tests of the Postgres cluster and consider automation that injects failures continually
   • Consider adding arbitrary restrictions to game day exercises, such as automation that has failed and a problem that cannot be diagnosed, so engineers practise bringing the service back another way

   Lessons

   A rule that two resources must not run together does not behave like the opposite of a rule that they must run together, and in some failure conditions the difference stopped Pacemaker from promoting a replica. A misconfiguration that shows only when two processes crash at almost the same time will not appear in basic tests or daily operations, only through fault injection. After two years in which the automation had handled every failure, the team's skill at controlling the cluster by hand had faded, and a runbook unused for years would likely have been out of date.

   View case
3. IL-0391 GoCardless API requests timed out for about 15 seconds while a Postgres migration waited for an AccessExclusive lock

   GoCardless2015

   Symptoms

   Just after a planned Postgres schema migration started, GoCardless received alerts that API requests were timing out. The API was down for about 15 seconds, until the migration went through and the API came back.

   Lock contention or deadlockService outage

   Trigger

   A deploy with a schema migration that renamed the empty invoicing tables and dropped and recreated their foreign key constraints, which referenced some of the most heavily used tables.

   Root cause

   Faulty deployment

   Component

   DatabaseSchema migration

   Contributing factor

   Unverified assumption

   Detection

   Alerts about API requests timing out began just after the schema migration started.

   Lessons

   Avoiding renames of tables in use, table rewrites under an exclusive lock and expensive work while holding one was not enough for GoCardless: a short DDL statement can still cause downtime by waiting in the lock queue behind a long-running query. GoCardless recommends removing long-running queries and transactions from the application and running analytics on an asynchronously updated replica, setting log_min_duration_statement and log_lock_waits to find such queries, setting lock_timeout in migration scripts so a deploy aborts rather than taking the application down, splitting schema changes so locks are held for shorter times, and keeping Postgres up to date. Developers with a console on the production database can cause the same downtime by leaving a transaction open.

   View case

By root cause3