Cases2 cases

1 to 2 of 2

1. IL-0346 Render's routing service returned intermittent 404s for web services and static sites in Oregon

   Render2025

   Symptoms

   On 5 November 2025, some web services and static sites hosted on Render in the Oregon region answered part of their requests with 404 responses. Every newly created service in the affected clusters got 404 responses, and updates to existing services were slow to take effect.

   High error rate

   Trigger

   On 5 November 2025 Render cleaned up feature flags that no longer appeared in code, including a flag that the metadata service still read, so the metadata service went back to its older, less memory-efficient way of sending data to the routing service.

   Root cause

   Configuration change

   Component

   Cloud load balancerService discovery

   Contributing factor

   Partial rolloutNo health probesAutomation amplification

   Remediation

   • Increased the memory available to the metadata service, a change later reverted
   • Re-enabled the feature flag for the more efficient data transfer between the routing and metadata services for a time; the flag was later removed
   • Deployed the metadata service so that it no longer relied on the feature flag

   Action items

   • Enhanced monitoring of the metadata service so that this failure mode raises an alert
   • Change feature flag practice so that a flag cannot be removed while it is still being evaluated
   • Keep a routing service instance out of traffic if it never loaded state from the metadata service

   View case
2. IL-0371 Half of new connections to Render Postgres databases with point-in-time recovery failed while their backup jobs ran

   Render2023

   Symptoms

   After Point-in-time Recovery (PITR) was released to eligible new Render Postgres databases on 21 April 2023, 50% of new connections to PITR-enabled databases failed while a PITR backup job was running. A backup takes 15 to 45 minutes depending on database size, and existing connections, such as those held through connection pooling, kept working.

   Intermittent connection failuresMisrouted requests

   Trigger

   Point-in-time Recovery was released to eligible new databases on 21 April 2023, and its backup job carried an incorrect networking configuration.

   Root cause

   Faulty deployment

   Component

   DatabaseDatabase backups and snapshotsService discovery

   Contributing factor

   –

   Remediation

   • Disabled backups for affected databases until they could be scheduled in low-traffic hours
   • Changed the backup job's network labels so the routing layer no longer picks it up as a database

   Action items

   • Tighten the process for sharing incident updates so problems are not declared resolved too early
   • Use more specific network labels for the database routing layer so ineligible IPs are never added
   • Run a skipped asynchronous job in the next scheduled window instead of immediately
   • Use open source benchmarking software to reproduce issues customers notice

   View case

By root cause2