Incident Library

IL-0219 GoCardless · 2017-10-10database

GoCardless's API and Dashboard failed every request for 1 hour 50 minutes when Pacemaker did not promote a replica

GoCardless keeps its most critical data, such as payment records and merchant and customer details, in a Postgres cluster of 3 nodes: a primary, a synchronous replica and an asynchronous replica. Pacemaker runs on each node, promotes the synchronous replica when the primary fails, and manages the virtual IP address (VIP) that the Ruby on Rails applications use to reach the primary. Normally such a failover would cost 1 or 2 minutes. When the primary's disk array failed on 10 October, Pacemaker did not promote the synchronous replica, and the team spent about an hour trying to make it do so. At 16:18 BST the team put the cluster into maintenance mode one last time, configured the synchronous replica as the primary and started Postgres by hand. Because Pacemaker in maintenance mode no longer moved the VIP, the backend applications were reconfigured to connect to the new primary's actual IP address; the manually promoted primary was working at 16:46 and monitoring confirmed the API and Dashboard were back at 16:59.

Afterwards the team bootstrapped a third node as an asynchronous replica, added a second VIP that the infrastructure team would move by hand in a failover, and found no evidence of data corruption. Using a version of the cluster that runs in Docker containers, one subteam scripted failures until it reproduced the production behaviour. Three conditions were all needed: default-resource-stickiness set to a non-zero value so Pacemaker avoids moving running resources; a Backup VIP, added so backups always connect to a replica, with a colocation rule of -INF preference against the Postgres primary, and running at the time on the synchronous replica; and a Postgres subprocess crashing on the synchronous replica almost at the same time as the primary failed. The invalid WAL file turned out to be a red herring: the failing primary had archived it to the backup server as a final act, Postgres discarded it, and the replica already had those writes through synchronous streaming replication. Writing the rule the other way round, an INF preference between the Backup VIP and a replica, let the cluster promote the synchronous replica under the same failures.

Because so much manual intervention had gone into the old cluster, the team provisioned a new Pacemaker-managed cluster and moved to it by adapting its zero-downtime failover procedure, in which PgBouncer on each node, behind its own VIP, pauses incoming queries while the Postgres VIP moves to a new primary and then resumes them. After practice runs, the migration was done in an announced maintenance window without problems, the old cluster was decommissioned and the incident closed.

Severity
–
Scope
GoCardless API and Dashboard: every request failed with an error for 1 hour and 50 minutes; no data corruption found
Users affected
–
Region
–
Time to detect
–
Time to resolve
1h 50m

Symptoms

At 15:09 BST on 10 October 2017, GoCardless's monitoring detected a total outage of its API and Dashboard: every request failed and returned an error. By 15:11 engineers saw evidence of a disk array failure on the primary Postgres node, but the Pacemaker cluster had not failed over to the synchronous replica as it should have.

At 15:17 the broken primary was powered off, in the belief that with only the two replicas online Pacemaker would promote one of them; it did not. At 15:18 clearing the error counts with crm resource cleanup, to make Pacemaker rediscover the state of the Postgres instances, had no effect. For the next hour engineers tried other ways to promote a primary, ending with putting the cluster into maintenance mode (crm configure property maintenance-mode=true), removing the flag in recovery.conf that makes Postgres a replica, and taking the cluster out of maintenance mode. Every time, Pacemaker brought the replica back with its original recovery.conf, leaving no primary.

The logs examined afterwards showed that the RAID controller had lost 3 disks of the array at the same moment, after which every read and write against it failed and the Linux kernel set the filesystem to read-only. Pacemaker had correctly seen that Postgres on the primary was unhealthy and kept trying to promote a new primary, but each time could not decide where it should run. On the synchronous replica one Postgres subprocess had crashed at about the same time, making Postgres restart, and afterwards the replica kept failing to restore a WAL file through its restore_command with a message that the file was invalid. That message later turned out to have nothing to do with the failed promotion.

Symptom Service outageFailover failure

Root cause

Root cause Hardware failure

Component DatabaseDatabase failover manager
Contributing factor Overlapping failuresUnverified assumption
Trigger
The RAID controller of the primary Postgres node lost 3 disks of its array at once, so all reads and writes on the primary failed.

Detection

GoCardless's monitoring detected the total outage of the API and Dashboard at 15:09 BST, and engineers began investigating.

Remediation

Action items

Lessons

A rule that two resources must not run together does not behave like the opposite of a rule that they must run together, and in some failure conditions the difference stopped Pacemaker from promoting a replica. A misconfiguration that shows only when two processes crash at almost the same time will not appear in basic tests or daily operations, only through fault injection. After two years in which the automation had handled every failure, the team's skill at controlling the cluster by hand had faded, and a runbook unused for years would likely have been out of date.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Database, Database failover manager among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.