Incident Library

IL-0345 Render · 2020-12-06networking

Many Render services in the Oregon region returned 503 errors for about four hours

Render's load balancing algorithm for HTTP requests is meant to handle a service that holds requests by rejecting traffic to that service, so that it cannot exhaust TCP connections on a load balancer shared with other services. When a customer API with one of the highest request rates in Oregon started holding requests until they timed out, it exposed a bug in that algorithm, and the load balancers also rejected some requests to other services with 503. Render mitigated the impact until the root cause was found, then rolled out a fix to all of its load balancers.

Severity
–
Scope
many Render services in the Oregon region and the Render Dashboard
Users affected
–
Region
Oregon
Time to detect
–
Time to resolve
3h 59m

Symptoms

Between 13:34 and 17:33 PST on Sunday 6 December 2020, many Render services in the Oregon region returned 503 (Service Unavailable) errors at an elevated rate, and the Render Dashboard was affected. Render's monitoring alerted the team at once. The status page reported upstream load balancers having trouble connecting to the rest of the infrastructure, and after an apparent fix the connectivity error rates rose again before the final fix.

Symptom High error rate

Root cause

Root cause Software defect

Component Cloud load balancer
Contributing factor Excessive blast radiusGap in automated tests
Trigger
At 13:34 PST a customer API hosted in Oregon, one of the services with the highest request rates in the region, began holding incoming requests until they timed out.

Detection

Render's monitoring system notified the team immediately.

Remediation

Action items

Lessons

–

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Cloud load balancer among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.