Incident Library

IL-0307 Supabase · 2025-01-29database

Supabase's Postgres connection pooler Supavisor 2.0 dropped connections for about 80% of us-east-1 tenants

Supavisor manages Postgres connections for Supabase. Supabase scheduled a major upgrade to Supavisor 2.0 across 10 regions; the new version had already run in 9 smaller regions over the previous weeks. On January 29, 2025 the maintenance began at 13:00 UTC and Supavisor 2.0 was deployed to 9 regions with minimal downtime, needing only client connection restarts. The us-east-1 deployment completed at 14:46 UTC with clients reconnected and throughput stable. At 16:05:50 UTC an :ets.select_delete operation failed and corrupted an ETS table; process registry lookups in Syn and Cachex that rely on it began failing, and at 16:06:00 UTC the processes managed by the registry crashed, dropping connections for about 80% of tenants. The incident was declared at 16:11 UTC and a rollback started. It completed at 16:53 UTC, and the previous version restored normal operation. During the incident Supabase advised users to fall back to direct database connections.

Severity
–
Scope
Postgres connections through the Supavisor pooler and Supabase Storage connectivity in us-east-1
Users affected
about 80% of tenants in us-east-1
Region
us-east-1
Time to detect
6m
Time to resolve
48m

Symptoms

At 16:05 UTC on January 29, 2025, most clients connected through Supavisor, Supabase's Postgres connection pooler, in us-east-1 were disconnected and could not reconnect reliably. Connections dropped for about 80% of tenants, and Supabase Storage connectivity also failed until the pooler was fixed.

Inside Supavisor 2.0, an :ets.select_delete operation failed at 16:05:50 UTC and corrupted an ETS (Erlang Term Storage) table. From 16:05:56 UTC, process registry lookups in the Syn and Cachex libraries, which are backed by ETS, failed and kept tenants from reconnecting, and at 16:06:00 UTC the processes managed by that registry crashed.

Symptom Connection resetsIntermittent connection failures

Root cause

Root cause Faulty deployment

Component Database proxy and connection poolerDatabase
Contributing factor –
Trigger
The planned upgrade of Supavisor to version 2.0 in us-east-1, which completed at 14:46 UTC.

Detection

–

Remediation

Action items

Lessons

Rollback was too slow and the pooler was not resilient enough to unforeseen failures under load; future work focuses on faster rollbacks to reduce the impact of similar failed deployments.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Database proxy and connection pooler, Database among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.