Incident Library

IL-0487 Autumn · 2026-03-15database

Autumn's Postgres CPU spiked and about 10% of API requests failed after a collation change left the main query unindexed

Autumn was optimising its most frequent queries and found that several indexes were not used because of a collation mismatch: customers.internal_id uses COLLATE "C" for KSUID ordering, while foreign key columns such as customer_entitlements.internal_customer_id used the default collation, and Postgres cannot use indexes across that mismatch. The most frequent query took about 150 ms instead of under 10 ms. Changing the collation on customer_entitlements first cut CPU by 30%, so the team decided to align four tables in the order customer_entitlements, customer_products, entities, customers. After the first two tables were changed, the main query lost the indexes it needed, database CPU spiked and queries failed.

Rolling back with ALTER TABLE deadlocked repeatedly against active queries, and a code-level cast did not help quickly enough. At 15:48, with help from PlanetScale support, the team shut down all application connections, reverted the collation changes, ran ANALYZE to rebuild the planner statistics and restarted. Queries returned to normal at once and CPU levelled off, and full recovery in both regions was confirmed at 16:15.

Severity
–
Scope
Autumn API: about 10% of requests failed from 15:00 to 16:15 GMT
Users affected
–
Region
both regions
Time to detect
5m
Time to resolve
1h 15m

Symptoms

On Sunday March 15, 2026 at about 15:00 GMT, right after collation changes were applied to the customer_entitlements and customer_products tables, database CPU on Autumn's production Postgres spiked and queries began to fail. About 10% of API requests to Autumn failed until 16:15 GMT.

The team identified the cause at 15:05 and tried to roll the collation back with ALTER TABLE. ALTER TABLE needs an exclusive lock on the table, and active queries always held locks on it, so the rollback ran into repeated deadlocks. At 15:20 a code change that cast the collation explicitly in the main customer query was pushed to get the indexes used again without a schema rollback, but with the database under heavy load it did not take effect fast enough.

Symptom Database overloadHigh error rateLock contention or deadlock

Root cause

Root cause Configuration change

Component Database
Contributing factor Ineffective rollback
Trigger
A migration that aligned column collations across related tables changed the collation of customer_entitlements and then customer_products on the production database.

Detection

–

Remediation

Action items

Lessons

–

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Database among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.