Incident Library

IL-0346 Render · 2025-11-05networking

Render's routing service returned intermittent 404s for web services and static sites in Oregon

Render's routing service depends on a metadata service for information about the user services it routes HTTP traffic to. When a routing instance starts, and on occasional reconnects, it requests a large volume of data from the metadata service. Earlier in 2025 Render deployed a memory optimisation for this transfer behind a feature flag. In late October the flag was removed from code and redeployed, but the metadata service was not redeployed and still depended on the flag. On 5 November the cleanup of unreferenced feature flags removed it, the metadata service reverted to the less efficient transfer, ran out of memory and crashed.

The routing service is designed to keep serving from its last known state when the metadata service is down, but new routing instances that never loaded their initial state were wrongly put into service and returned 404. In the first period the metadata service was crashing in two clusters and only a small share of routing instances was affected. In the second period a surge of HTTP requests scaled up the routing service, and all new instances returned 404, so about half the requests in the affected cluster failed. Render gave the metadata service more memory, re-enabled the feature flag for a time, and then deployed the metadata service without its dependency on the flag. The status page moved to monitoring at 20:48 UTC and the incident was marked resolved at 21:52 UTC.

Severity
–
Scope
Web services and static sites in two Oregon clusters returned intermittent 404 responses, about 10% of requests in one cluster in the first period and about 50% in the affected cluster in the second; newly created services got 404, updates to existing services propagated slowly, and recently deployed or waking free tier services were affected
Users affected
Some customers hosting web services and static sites in the Oregon region
Region
Oregon
Time to detect
–
Time to resolve
1h 55m

Symptoms

On 5 November 2025, some web services and static sites hosted on Render in the Oregon region answered part of their requests with 404 responses. Every newly created service in the affected clusters got 404 responses, and updates to existing services were slow to take effect.

The 404 responses came in two periods. From 10:39 AM to 11:25 AM PST two Render clusters were slightly degraded: one returned a negligible number of 404 responses and the other returned 404 for about 10% of requests. From 11:59 AM to 12:34 PM PST the degradation was larger, and about 50% of all requests to services in the affected cluster received a 404. Free tier services that had just been deployed or were waking from sleep were also hit.

Behind the 404 responses, the metadata service that tells Render's routing service about the user services it routes to was running out of memory and crashing. The routing service is designed to keep serving traffic from its last known state when the metadata service is down, but newly started routing instances that could not load their initial state from the metadata service were still sent requests and answered them with 404. In the second period a large rise in HTTP requests to the affected cluster made the routing service scale up, and every new routing instance returned 404.

Symptom High error rate

Root cause

Root cause Configuration change

Component Cloud load balancerService discovery
Contributing factor Partial rolloutNo health probesAutomation amplification
Trigger
On 5 November 2025 Render cleaned up feature flags that no longer appeared in code, including a flag that the metadata service still read, so the metadata service went back to its older, less memory-efficient way of sending data to the routing service.

Detection

–

Remediation

Action items

Lessons

–

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Cloud load balancer, Service discovery among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.