Cases1 case

1 to 1 of 1

1. IL-0492 Hosted Graphite's self-hosted load balancers hit their connection limit in an AWS outage, timing out its site and APIs

   Hosted Graphite2018

   Symptoms

   On a Friday night at 23:28 UTC, Hosted Graphite's monitoring showed its website intermittently becoming unavailable: periodic checks of the website and of the graph rendering API were timing out. Traffic dropped across all ingestion protocols at the same time.

   Intermittent timeoutsIntermittent connection failures

   Trigger

   An AWS connectivity problem in us-east1 and us-west2 slowed down connections from AWS hosts in those regions to Hosted Graphite's load balancers without breaking them.

   Root cause

   Capacity exhaustion

   Component

   On-premises load balancer

   Contributing factor

   Circular dependency

   Detection

   Monitoring reported the website intermittently unavailable at 23:28 UTC, when periodic checks of the website and the graph rendering API started timing out.

   Action items

   • Enforce stricter client timeouts
   • Raise the load balancers' connection limits
   • Use the new visualizations built from load balancer logs when troubleshooting the load balancing layer

   Lessons

   An outage at a provider that did not host the service's website still made the service unavailable, here because slow clients in the affected regions used up the connection limit of the load balancers that served everyone.

   View case

By root cause1