IL-0317 Clerk · 2026-02-19database

Clerk's Postgres slowed after an automatic analyze flipped a query plan, and over 95% of requests got 429

The Postgres query planner chooses how to run a query from statistics that it collects from a sample of rows, and it re-analyzes tables regularly as data is added. In one column of the affected table 99.9996% of values were NULL. When the automatic analyze ran, its sample was too small and found only NULLs, so the planner concluded that 100% were NULL and planned the query as if one part would return no non-null rows, when it actually returned over 17,000. The query runs often enough that traversing those 17,000 rows each time took nearly all database resources, request handlers queued, and most requests were shed with status 429.

The team first suspected a customer's traffic spike and blocked it without effect. At 17:08 UTC a new failover that generates session tokens outside the core Session API was enabled, reducing backend load; it had been built and tested over the previous months but had no production triggers yet, and from 17:10 many users could reach customer applications again. At 17:25 the plan flip was identified, and at 17:27 ANALYZE was re-run manually on the table, the query returned to its earlier plan and the database began to recover. Systems were stable at 17:43, when the failover was disabled, and stable performance across the system was confirmed at 18:06. Right after the incident Clerk increased the statistics target on the table, and that evening refactored the query so the planner could plan it deterministically.

Severity: –
Scope: Over 95% of Clerk traffic returned 429 during an outage of about 90 minutes; sign in and account management APIs were unavailable; session tokens were served by a failover from 17:10 UTC
Users affected: Clerk customers and the users of their applications
Region: –
Time to detect: 4m
Time to resolve: 1h 30m

Symptoms

At 8:11 AM PST (16:11 UTC) on 19 February 2026, Clerk's database performance dropped at once. Slow database queries made request handlers queue, and over 95% of traffic was answered with 429 without being handled; the small share of requests that reached the database returned 200, but extremely slowly.

The incident was reported and escalated internally at 16:15 UTC. At 16:32 the team found a customer whose traffic spiked at the same time as the outage and suspected that load had triggered the database problems. Blocking that customer's traffic at 16:35 did not help, and by 16:50 the team found that the customer's overly aggressive retry mechanism had caused the spike, retrying failures that started inside Clerk's infrastructure.

The automatic failover for Clerk's Session API did not trigger, because the database was still online, only degraded. Even after a newer failover for session tokens was enabled at 17:08, sign in and account management APIs stayed unavailable.

Symptom Service outage Database overload High latency

Root cause

Root cause Query plan regression

Component Database Query planner and table statistics

Contributing factor Inefficient query Monitoring blind spot

Trigger: A routine automatic ANALYZE on one table sampled only NULL values in a column, and the Postgres planner switched a frequently run query to an inefficient plan.

Detection

The incident was reported and escalated internally at 16:15 UTC, and the team started a group call.

Remediation

Enabled a new failover that generates session tokens outside the core Session API
Re-ran ANALYZE manually on the affected table, which put the query back on its earlier plan
Disabled the failover once the database was back to healthy levels
Increased the statistics target on the table so the planner samples more rows
Refactored the query so the planner could plan it deterministically

Action items

Add dedicated alerting for database query plan flips
Train the reliability team to enable the new session token failover and add automatic triggers for it
Audit all queries for others at risk of inefficient plans from unstable statistics
Name a dedicated communications lead during incidents
Post status page updates at a regular cadence even when nothing has changed
Make status page severity labels match the impact on customers
Cross-post incident updates to social channels

Lessons

Clerk says a query plan flip can cause sudden, severe degradation and needs to be detected directly instead of through downstream symptoms; relying on those symptoms led the team to blame a customer's traffic pattern. Customers also told Clerk that status page updates came too rarely, the first update came too late and the initial severity label did not match the impact.

Sources

https://clerk.com/blog/2026-02-19-system-outage-postmortem · blog · archived copy

Related cases

–

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Database, Query planner and table statistics among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0317
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0317/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.