Incident Library

IL-0371 Render · 2023-04-21databasenetworking

Half of new connections to Render Postgres databases with point-in-time recovery failed while their backup jobs ran

Render Postgres databases are reached through a stable internal service name such as dpg-123abc, which the cluster keeps pointed at the database's current IP address. The daily PITR backup job, which keeps an accurate restore point for each database, was given networking labels similar to the database's own, so the routing layer added the backup job's IP address to the database's service name as a second target. The backup job ran no database, so new connections routed to it failed, which made half of new connections fail while a backup was running. Render first disabled backups on the affected databases. When backups were turned back on, the scheduler for asynchronous jobs immediately ran a backup that it counted as a recently missed run, which caused downtime during business hours. Render changed the backup job's network labels so the routing layer no longer treats it as a database endpoint, and the incident was marked resolved on 27 April 2023.

Severity
–
Scope
New connections to PITR-enabled Render Postgres databases while a PITR backup job was running; existing connections were unaffected
Users affected
–
Region
–
Time to detect
–
Time to resolve
–

Symptoms

After Point-in-time Recovery (PITR) was released to eligible new Render Postgres databases on 21 April 2023, 50% of new connections to PITR-enabled databases failed while a PITR backup job was running. A backup takes 15 to 45 minutes depending on database size, and existing connections, such as those held through connection pooling, kept working.

Render engineers had not seen this kind of failure before. The engineers noticed a change in memory behaviour, specifically the active page cache, and tried to reproduce it, but that attempt left out the fact that new connections were failing, so finding the root cause took time.

Backups were then disabled for affected databases until they could be scheduled in low-traffic hours. When a backup was enabled and rescheduled, the job ran at once because a recent run had been missed, and this caused another outage during business hours. Some early status updates said the problem was resolved before it was, and some users saw the failures come back.

Symptom Intermittent connection failuresMisrouted requests

Root cause

Root cause Faulty deployment

Component DatabaseDatabase backups and snapshotsService discovery
Contributing factor –
Trigger
Point-in-time Recovery was released to eligible new databases on 21 April 2023, and its backup job carried an incorrect networking configuration.

Detection

–

Remediation

Action items

Lessons

–

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Database, Database backups and snapshots, Service discovery among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.