Cases2 cases

1 to 2 of 2

1. IL-0346 Render's routing service returned intermittent 404s for web services and static sites in Oregon

   Render2025

   Symptoms

   On 5 November 2025, some web services and static sites hosted on Render in the Oregon region answered part of their requests with 404 responses. Every newly created service in the affected clusters got 404 responses, and updates to existing services were slow to take effect.

   High error rate

   Trigger

   On 5 November 2025 Render cleaned up feature flags that no longer appeared in code, including a flag that the metadata service still read, so the metadata service went back to its older, less memory-efficient way of sending data to the routing service.

   Root cause

   Configuration change

   Component

   Cloud load balancerService discovery

   Contributing factor

   Partial rolloutNo health probesAutomation amplification

   Remediation

   • Increased the memory available to the metadata service, a change later reverted
   • Re-enabled the feature flag for the more efficient data transfer between the routing and metadata services for a time; the flag was later removed
   • Deployed the metadata service so that it no longer relied on the feature flag

   Action items

   • Enhanced monitoring of the metadata service so that this failure mode raises an alert
   • Change feature flag practice so that a flag cannot be removed while it is still being evaluated
   • Keep a routing service instance out of traffic if it never loaded state from the metadata service

   View case
2. IL-0345 Many Render services in the Oregon region returned 503 errors for about four hours

   Render2020

   Symptoms

   Between 13:34 and 17:33 PST on Sunday 6 December 2020, many Render services in the Oregon region returned 503 (Service Unavailable) errors at an elevated rate, and the Render Dashboard was affected. Render's monitoring alerted the team at once. The status page reported upstream load balancers having trouble connecting to the rest of the infrastructure, and after an apparent fix the connectivity error rates rose again before the final fix.

   High error rate

   Trigger

   At 13:34 PST a customer API hosted in Oregon, one of the services with the highest request rates in the region, began holding incoming requests until they timed out.

   Root cause

   Software defect

   Component

   Cloud load balancer

   Contributing factor

   Excessive blast radiusGap in automated tests

   Detection

   Render's monitoring system notified the team immediately.

   Remediation

   • Rolled out a fix to the load balancing algorithm across all load balancers

   Action items

   • Add mechanisms that isolate services from each other in the routing and load balancing layers
   • Increase test coverage of the load balancing code
   • Add synthetic failure states to the continuous integration test suites
   • Put more focus on chaos engineering

   View case

By root cause2