IL-0492 Hosted Graphite · 2018-03-01 (published)networkingaws

Hosted Graphite's self-hosted load balancers hit their connection limit in an AWS outage, timing out its site and APIs

Hosted Graphite runs its own load balancers outside AWS and uses AWS Route53 health checks to keep only healthy load balancers in the DNS results of hostedgraphite.com. Many Hosted Graphite users are hosted in AWS themselves. During an AWS connectivity problem in us-east1 and us-west2, connections from AWS hosts in those regions became very slow without breaking: a heatmap built from the load balancer logs showed AWS hosts, mainly EC2 instances, taking up to 5 seconds to send a full HTTP request, while other ISPs were largely unaffected. The number of active connections to the load balancers rose sharply and then flatlined for exactly the period of the incident, even though the number of requests had fallen.

The slow AWS clients held every available connection until the load balancers reached their connection limit. New connections from everywhere else were queued and could time out before they were processed, which explained the slow SSL handshakes and the failures of the website, the render API and the HTTP API from every location. Traffic started to recover slowly when AWS resolved its incident at 01:46 UTC. Afterwards Hosted Graphite enforced stricter client timeouts and raised the load balancers' connection limits.

Severity: –
Scope: Hosted Graphite website, graph rendering API and HTTP ingestion API intermittently unavailable from all locations; ingestion from AWS us-east1 and us-west2 reduced; some internal services timed out
Users affected: –
Region: global
Time to detect: –
Time to resolve: –

Symptoms

On a Friday night at 23:28 UTC, Hosted Graphite's monitoring showed its website intermittently becoming unavailable: periodic checks of the website and of the graph rendering API were timing out. Traffic dropped across all ingestion protocols at the same time.

Internal and external canaries showed ingestion affected from some AWS regions, us-east1 and us-west2 in particular, while ingestion through the HTTP API was affected from every location. Some internal services that are not user-facing also reported timeouts. The internal services in trouble relied on S3, integrations with AWS dependencies such as Heroku were badly affected, and the AWS status page confirmed connectivity issues in us-east1 and us-west2. That did not explain why the website, which is neither hosted in AWS nor near those regions, was failing.

The team then suspected the Route53 health checks that remove an unhealthy self-hosted load balancer from the DNS results of hostedgraphite.com, since the health check logs reported failures from many locations. The SRE chatbot that would normally disable them depended on Slack's IRC gateway, which the AWS outage had also affected, so the health checks were disabled by hand. That did not fix anything.

The load balancer logs then showed failures to complete SSL handshakes for IP addresses outside the normal AWS ranges, and tests from several locations confirmed that SSL handshakes took unacceptably long wherever they came from.

Symptom Intermittent timeouts Intermittent connection failures

Root cause

Root cause Capacity exhaustion

Component On-premises load balancer

Contributing factor Circular dependency

Trigger: An AWS connectivity problem in us-east1 and us-west2 slowed down connections from AWS hosts in those regions to Hosted Graphite's load balancers without breaking them.

Detection

Monitoring reported the website intermittently unavailable at 23:28 UTC, when periodic checks of the website and the graph rendering API started timing out.

Remediation

–

Action items

Enforce stricter client timeouts
Raise the load balancers' connection limits
Use the new visualizations built from load balancer logs when troubleshooting the load balancing layer

Lessons

An outage at a provider that did not host the service's website still made the service unavailable, here because slow clients in the affected regions used up the connection limit of the load balancers that served everyone.

Sources

https://blog.hostedgraphite.com/blog/spooky-action-at-a-distance-how-an-aws-outage-ate-our-load-balancer · blog · archived copy

Related cases

IL-0262 Requests through one GitHub datacenter failed or slowed after its load balancer nodes ran out of network flowsGitHub · 2026
same root cause · shared On-premises load balancer
IL-0455 Calls from an AWS service through NAT Gateway to an on-premises service returned 5xx errors every busy weekendWoowa Brothers · 2020
same root cause · shared On-premises load balancer

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, On-premises load balancer among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0492
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0492/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.