Incident Library

IL-0391 GoCardless · 2015-05-27 (published)database

GoCardless API requests timed out for about 15 seconds while a Postgres migration waited for an AccessExclusive lock

In late January, GoCardless was building invoicing for its Pro product and had to rename tables that were already deployed. The tables were empty and no production code used them, and after the rename the team dropped and recreated their foreign key constraints so the names were up to date, expecting no long-held lock because there was no data to validate.

To add a foreign key constraint, Postgres takes AccessExclusive locks on both the table with the constraint and the table it references while it adds the triggers that enforce the constraint, and this applies even with NOT VALID. A lock that cannot be acquired waits in a queue, and every lock that conflicts with it queues behind it; because AccessExclusive conflicts with every other lock type, a waiting AccessExclusive lock blocks all other operations on that table. A long-running read query that happened to be running on one of the heavily used referenced tables held an AccessShare lock, the ALTER TABLE waited behind it, and the API's read and write queries on that table piled up behind the ALTER TABLE until clients timed out. The ALTER TABLE itself was fast, and the API recovered once it went through after about 15 seconds.

Severity
–
Scope
The GoCardless API stopped answering requests for about 15 seconds
Users affected
–
Region
–
Time to detect
–
Time to resolve
–

Symptoms

Just after a planned Postgres schema migration started, GoCardless received alerts that API requests were timing out. The API was down for about 15 seconds, until the migration went through and the API came back.

The same migrations, run afterwards against a backup of the database from earlier that day, finished in a few hundred milliseconds. Common advice on zero-downtime schema changes did not explain the outage; the explanation came from an old thread on the Postgres mailing list and a blog post that went into more depth.

Symptom Lock contention or deadlockService outage

Root cause

Root cause Faulty deployment

Component DatabaseSchema migration
Contributing factor Unverified assumption
Trigger
A deploy with a schema migration that renamed the empty invoicing tables and dropped and recreated their foreign key constraints, which referenced some of the most heavily used tables.

Detection

Alerts about API requests timing out began just after the schema migration started.

Remediation

–

Action items

–

Lessons

Avoiding renames of tables in use, table rewrites under an exclusive lock and expensive work while holding one was not enough for GoCardless: a short DDL statement can still cause downtime by waiting in the lock queue behind a long-running query. GoCardless recommends removing long-running queries and transactions from the application and running analytics on an asynchronously updated replica, setting log_min_duration_statement and log_lock_waits to find such queries, setting lock_timeout in migration scripts so a deploy aborts rather than taking the application down, splitting schema changes so locks are held for shorter times, and keeping Postgres up to date. Developers with a console on the production database can cause the same downtime by leaving a transaction open.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Database, Schema migration among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.