Incident Library

IL-0483 Replicate · 2023-05-11database

Replicate's website and API returned slow responses and HTTP 500s for two hours as PostgreSQL connections ran out

Replicate's primary datastore is a PostgreSQL database, and starting a prediction depended on it directly. To keep predictions working during database maintenance or short outages, Replicate was adding caches and queues, and over 9 and 10 May it rolled out, behind feature flags, API service changes that wrote prediction state updates to the database asynchronously, outside the request handlers. A little after 14:00 UTC on 10 May the database primary was resized in just over a minute, and most users saw no disruption.

The asynchronous path removed about 100-150ms from creating a prediction, so two updates to the same prediction that used to arrive that far apart were now processed almost at the same time, and both tried to INSERT a row with the same unique prediction ID. Only one can succeed, and the queries contend for the unique index lock while the database decides, which makes all of them slower. The code fell back to updating the existing row when its INSERT failed, so users saw nothing at first, but slower INSERTs meant more parallel queries and more lock contention. A little after 05:00 UTC on 11 May normal traffic variation tipped this over, INSERT latency rose to tens of seconds, and from about 05:45 UTC the database connections were exhausted. After Crunchy Data helped rule out the database, Replicate switched all asynchronous prediction update features off with a feature flag at 07:31 UTC, and the database recovered within a few minutes.

Severity
–
Scope
replicate.com website and API: slow responses and HTTP 500 errors, predictions failing to run; webhook users saw relatively little disruption
Users affected
many customers
Region
–
Time to detect
1m
Time to resolve
2h

Symptoms

From 05:45 UTC on 11 May 2023, for about two hours, many Replicate customers got slow responses or HTTP 500 errors from the API and the replicate.com website and had trouble running predictions. Monitoring paged an engineer at 05:46 UTC, a minute after the database connection pool became fully saturated.

In the PostgreSQL database, INSERT latency jumped within a few minutes from about 1 millisecond to tens of seconds. More queries ran in parallel, each with its own database connection, until the maximum permitted number of connections was used up and new queries could not run at all. The website failed intermittently and was down more than it was up. Customers who received prediction updates by webhook instead of polling the API saw little disruption.

The first suspicion was the resize of the database primary the day before: perhaps the new primary had slower disks, or an auxiliary job on the new server was using disk I/O. The new asynchronous prediction update code looked an unlikely cause, because it had been enabled for more than 18 hours without visible problems. Replicate's database provider, Crunchy Data, responded within minutes and helped rule out problems with the database itself.

Symptom High error rateHigh latencyLock contention or deadlock

Root cause

Root cause Faulty deployment

Component Database
Contributing factor Change with a delayed effect
Trigger
A little after 05:00 UTC on 11 May, normal variations in traffic pushed the stacking INSERT queries past their tipping point.

Detection

Monitoring detected the saturated database connection pool almost immediately and paged an engineer at 05:46 UTC.

Remediation

Action items

Lessons

The system ran in a degraded state that looked successful until it tipped into a full failure. The same asynchronous update work that triggered the outage also partly protected customers who used webhooks, and the stacked INSERTs came directly from a latency improvement: removing 100-150ms brought two updates of one prediction to the database almost together.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Database among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.