Cases2 cases

1 to 2 of 2

1. IL-0307 Supabase's Postgres connection pooler Supavisor 2.0 dropped connections for about 80% of us-east-1 tenants

   Supabase2025

   Symptoms

   At 16:05 UTC on January 29, 2025, most clients connected through Supavisor, Supabase's Postgres connection pooler, in us-east-1 were disconnected and could not reconnect reliably. Connections dropped for about 80% of tenants, and Supabase Storage connectivity also failed until the pooler was fixed.

   Connection resetsIntermittent connection failures

   Trigger

   The planned upgrade of Supavisor to version 2.0 in us-east-1, which completed at 14:46 UTC.

   Root cause

   Faulty deployment

   Component

   Database proxy and connection poolerDatabase

   Contributing factor

   –

   Remediation

   • Rolled Supavisor back to the previous version

   Action items

   • Cut rollback time from 40 minutes to under 90 seconds
   • Detect client disconnections within 2 minutes

   Lessons

   Rollback was too slow and the pooler was not resilient enough to unforeseen failures under load; future work focuses on faster rollbacks to reduce the impact of similar failed deployments.

   View case
2. IL-0444 Reader connections to Repro's Aurora MySQL cluster were cut with Lost connection errors every time ALTER TABLE ran

   Repro2024

   Symptoms

   After Repro moved to Aurora MySQL 5.7, running an ALTER TABLE in production, even one that finished in under a second, made the processes reading that table fail with `Lost connection to MySQL server during query`. MySQL client connections were cut right after the ALTER TABLE ran, and many application errors followed. By 8 January 2021 this had happened three times in a row.

   Connection resetsHigh error rate

   Trigger

   Running ALTER TABLE on the Aurora MySQL writer while readers were running queries that used the same table.

   Root cause

   Breaking change in an upgrade

   Component

   Database

   Contributing factor

   Unrepresentative staging

   Remediation

   • Made the applications retry once after a connection error, with a monkey patch in the Rails applications

   View case

By root cause2