IL-0251 Slack · 2020-05-12networkingaws

Most of Slack's HAProxy load balancers routed to a stale, shrinking set of webapp backends and served HTTP 503 errors

Slack sends requests to its webapp tier through a fleet of HAProxy instances behind a layer 4 load balancer. Consul provides service discovery, and consul-template renders the list of healthy webapp backends. To avoid reloading HAProxy for every change, a Slack program called haproxy-server-state-management reads that list and updates HAProxy through its Runtime API. HAProxy is configured with server templates that act as slots: N slots for webapp backends in each of M Availability Zone pools, N * M in total, which a few months earlier had been far more than Slack ever needed.

At about 8:30am that day a configuration change, rolled out as a percentage-based feature flag, triggered a longstanding performance bug and raised load on part of Slack's database infrastructure. The flag was rolled back and customer impact lasted three minutes, but slow database requests saturated webapp workers and autoscaling raised the instance count by 75%, the most webapp hosts Slack had ever run and slightly more than N * M. haproxy-server-state-management had a bug: it always tried to place new webapp instances in a slot before freeing the slots of instances that had gone away. With no empty slot it failed and exited early, so the running HAProxy instances stopped receiving state updates. Over the day the backend lists grew more and more stale. When the webapp tier scaled down at the end of the US business day, autoscaling removed the older instances first, and the backends still listed in HAProxy could no longer serve the demand.

Once the cause was known, a rolling restart of the HAProxy fleet resolved the outage. Slack found that its alerting for this failure was broken, which had gone unnoticed because the system had worked without changes for a long time. Slack was already moving its ingress load balancing to Envoy Proxy, whose xDS integration replaces this pipeline and is not exposed to this failure.

Severity: –
Scope: Slack requests through the HAProxy tier failed with HTTP 503 errors from 4:45pm Pacific
Users affected: –
Region: –
Time to detect: –
Time to resolve: –

Symptoms

At 4:45pm Pacific on 12 May 2020, Slack was alerted that it was serving more HTTP 503 errors than normal. A subset of the webapp fleet was under heavy load while the rest of the webapp instances were not.

The on-call engineer for the webapp tier first scaled up the webapp fleet by hand, which did not help at all. Investigation then looked at both webapp performance and the load balancer tier, and a few minutes later found that most HAProxy instances could only send requests to the webapp backends that had been running since that morning, which by then were a minority of the fleet. Only a few HAProxy instances provisioned recently had the correct backend list.

The backend state inside HAProxy had been going stale for about eight hours without any visible effect. The program that copies the host list from consul-template into HAProxy had been failing and exiting early, so as the webapp autoscaling group scaled up and down during the day, HAProxy's backends came to refer mainly to hosts that no longer existed. Alerting for exactly this situation was in place but was not working as intended.

Symptom High error rate Load imbalance Stale endpoint routing

Root cause

Root cause Software defect

Component HAProxy Service discovery

Contributing factor Automation amplification No alert on limit

Trigger: At the end of the US business day the webapp tier began its usual scale-down, and autoscaling terminated the oldest instances first, which were the only backends left in most HAProxy instances' state.

Detection

An alert for more HTTP 503 errors than normal. The alerting meant to catch stale HAProxy state was not working.

Remediation

Did a rolling restart of the HAProxy fleet

Action items

Move all ingress load balancing from HAProxy to Envoy Proxy, with endpoint discovery through an xDS control plane

Lessons

The alerting for this exact failure had stopped working, and nobody noticed because the HAProxy stack had run for a long time without needing changes, so few engineers worked with its monitoring and alerting. A slot limit that had been ample a few months earlier was reached after pandemic-driven growth and one large autoscaling event.

Sources

https://slack.engineering/a-terrible-horrible-no-good-very-bad-day-at-slack/ · blog · archived copy

Related cases

IL-0213 Roblox was down for 73 hours because key-value writes on its single Consul cluster slowed from under 300 ms to 2 secondsRoblox · 2021
same root cause · shared Service discovery
IL-0271 An empty etcd service entry crashed Discord's Elixir chat nodes, and Discord was down for most users for an hourDiscord · 2020 · sev1
same root cause · shared Service discovery
IL-0011 HAProxy ingress piled requests onto the same pods because a Go template visited the backend map in sorted pod IP orderSkyscanner · 2019
same root cause · shared HAProxy
IL-0587 Elastic Cloud proxies in AWS eu-west-1 flooded an overloaded ZooKeeper and ran out of memory, cutting cluster accessElastic · 2019
same root cause · shared Service discovery

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, HAProxy, Service discovery among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0251
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0251/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.