Cases1 case

1 to 1 of 1

1. IL-0345 Many Render services in the Oregon region returned 503 errors for about four hours

   Render2020

   Symptoms

   Between 13:34 and 17:33 PST on Sunday 6 December 2020, many Render services in the Oregon region returned 503 (Service Unavailable) errors at an elevated rate, and the Render Dashboard was affected. Render's monitoring alerted the team at once. The status page reported upstream load balancers having trouble connecting to the rest of the infrastructure, and after an apparent fix the connectivity error rates rose again before the final fix.

   High error rate

   Trigger

   At 13:34 PST a customer API hosted in Oregon, one of the services with the highest request rates in the region, began holding incoming requests until they timed out.

   Root cause

   Software defect

   Component

   Cloud load balancer

   Contributing factor

   Excessive blast radiusGap in automated tests

   Detection

   Render's monitoring system notified the team immediately.

   Remediation

   • Rolled out a fix to the load balancing algorithm across all load balancers

   Action items

   • Add mechanisms that isolate services from each other in the routing and load balancing layers
   • Increase test coverage of the load balancing code
   • Add synthetic failure states to the continuous integration test suites
   • Put more focus on chaos engineering

   View case

By root cause1