Incident Library
Applied filters ComponentCloud load balancerSymptomHigh error rateFactorGap in automated testsClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0345 Many Render services in the Oregon region returned 503 errors for about four hours
    Render2020
    Symptoms

    Between 13:34 and 17:33 PST on Sunday 6 December 2020, many Render services in the Oregon region returned 503 (Service Unavailable) errors at an elevated rate, and the Render Dashboard was affected. Render's monitoring alerted the team at once. The status page reported upstream load balancers having trouble connecting to the rest of the infrastructure, and after an apparent fix the connectivity error rates rose again before the final fix.

    High error rate
    Trigger

    At 13:34 PST a customer API hosted in Oregon, one of the services with the highest request rates in the region, began holding incoming requests until they timed out.

    Root cause
    Software defect
    Component
    Cloud load balancer
    Contributing factor
    Excessive blast radiusGap in automated tests
    Detection

    Render's monitoring system notified the team immediately.

    Remediation
    • Rolled out a fix to the load balancing algorithm across all load balancers
    Action items
    • Add mechanisms that isolate services from each other in the routing and load balancing layers
    • Increase test coverage of the load balancing code
    • Add synthetic failure states to the continuous integration test suites
    • Put more focus on chaos engineering
    View case
  2. IL-0338 Google Cloud HTTP(S) and TCP/SSL Proxy load balancers returned 502s or reset connections on 33% to 87% of requests
    Google Cloud2018
    Symptoms

    From 12:17 to 12:49 PDT on 17 July 2018, Google Cloud HTTP(S) Load Balancers answered a share of requests with 502 return codes, and connections to TCP/SSL Proxy Load Balancers were reset. The share of failed requests moved between 33% and 87% during those 32 minutes, and automated monitoring alerted Google engineers at 12:19.

    High error rateConnection resetsHigh latency
    Trigger

    A configuration change in the production environment intermittently set off a bug in a new feature of the second-layer Google Front Ends.

    Root cause
    Software defect
    Component
    Cloud load balancerCDN and edge network
    Contributing factor
    Gap in automated testsExcessive blast radius
    Detection

    Automated monitoring alerted Google's engineering team at 12:19 PDT, two minutes after the 502s began.

    Remediation
    • Reverted the configuration change at 12:44 PDT, which stopped the GFE restarts
    Action items
    • Add safeguards that disable features not yet in service
    • Harden the GFE testing stack against latent bugs in production binaries that make a task restart
    • Isolate the shards of GFE pools further from each other to reduce the scope of failures
    • Build a consolidated dashboard of all configuration changes for GFE pools
    View case
Check your cluster for this

These 2 cases involve Cloud load balancer and CDN and edge network. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Software defect2 cases

Component
Cloud load balancer2CDN and edge network1
Contributing factor
Excessive blast radius2Gap in automated tests2
Organisation
Google Cloud1Render1
  1. IL-0345 Many Render services in the Oregon region returned 503 errors for about four hours
    Render2020
  2. IL-0338 Google Cloud HTTP(S) and TCP/SSL Proxy load balancers returned 502s or reset connections on 33% to 87% of requests
    Google Cloud2018