Cases1 case

1 to 1 of 1

1. IL-0487 Autumn's Postgres CPU spiked and about 10% of API requests failed after a collation change left the main query unindexed

   Autumn2026

   Symptoms

   On Sunday March 15, 2026 at about 15:00 GMT, right after collation changes were applied to the customer_entitlements and customer_products tables, database CPU on Autumn's production Postgres spiked and queries began to fail. About 10% of API requests to Autumn failed until 16:15 GMT.

   Database overloadHigh error rateLock contention or deadlock

   Trigger

   A migration that aligned column collations across related tables changed the collation of customer_entitlements and then customer_products on the production database.

   Root cause

   Configuration change

   Component

   Database

   Contributing factor

   Ineffective rollback

   Remediation

   • Shut down all application connections to the database
   • Reverted the collation changes
   • Ran ANALYZE to rebuild the planner statistics and restarted

   Action items

   • Do all future column and schema migrations with the duplicate-dual-write-cutover pattern: add a new column, dual-write, backfill in small batches, create indexes concurrently, then cut over
   • Add a status page
   • Consider making the check and track SDKs fail open by default

   View case

By root cause1