Incident Library
Applied filters ComponentOn-premises load balancerFactorCircular dependencyClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0492 Hosted Graphite's self-hosted load balancers hit their connection limit in an AWS outage, timing out its site and APIs
    Hosted Graphite2018
    Symptoms

    On a Friday night at 23:28 UTC, Hosted Graphite's monitoring showed its website intermittently becoming unavailable: periodic checks of the website and of the graph rendering API were timing out. Traffic dropped across all ingestion protocols at the same time.

    Intermittent timeoutsIntermittent connection failures
    Trigger

    An AWS connectivity problem in us-east1 and us-west2 slowed down connections from AWS hosts in those regions to Hosted Graphite's load balancers without breaking them.

    Root cause
    Capacity exhaustion
    Component
    On-premises load balancer
    Contributing factor
    Circular dependency
    Detection

    Monitoring reported the website intermittently unavailable at 23:28 UTC, when periodic checks of the website and the graph rendering API started timing out.

    Action items
    • Enforce stricter client timeouts
    • Raise the load balancers' connection limits
    • Use the new visualizations built from load balancer logs when troubleshooting the load balancing layer
    Lessons

    An outage at a provider that did not host the service's website still made the service unavailable, here because slow clients in the affected regions used up the connection limit of the load balancers that served everyone.

    View case
Check your cluster for this

This case involves On-premises load balancer. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion1 case

Component
On-premises load balancer
Contributing factor
Circular dependency
Organisation
Hosted Graphite
  1. IL-0492 Hosted Graphite's self-hosted load balancers hit their connection limit in an AWS outage, timing out its site and APIs
    Hosted Graphite2018