IL-0562 Linode · 2022-12-13networking

Most Linodes in Linode's Dallas data center had intermittent latency and reachability loss from broadcast storms

In Linode's Dallas data center, broadcast traffic grew until it affected the hypervisors. Most of it came from ARP requests that got no answer, and each unanswered request led to more broadcasts, so the traffic cascaded and amplified into broadcast storms. The broadcast traffic caused intermittent latency and reachability loss for about 80-90% of the Linodes in the data center. The underlying cause of the storms was still under investigation when the postmortem was posted, with the other recent incidents in the Dallas data center taken into account.

At about 04:20 UTC a filter that stemmed the broadcasts was deployed to the fleet. With the broadcasts stemmed, the hosts regained CPU cycles and answered ARP requests consistently, the network recovered, and alerts cleared in bulk by 04:32. After more monitoring the incident was resolved at 06:45. The filter was first a temporary measure; it was tested further and made permanent on 13 December 2022 at 18:35 UTC.

Severity: –
Scope: Linodes in the Dallas data center had intermittent latency and reachability loss; Cloud Manager and the Linode API could be hard to reach
Users affected: about 80-90% of the Linodes in the Dallas data center
Region: US-Central (Dallas)
Time to detect: –
Time to resolve: 4h 27m

Symptoms

At about 00:05 UTC, Linode's monitoring systems alerted on reachability problems in the Dallas data center. About 80-90% of the Linodes there had intermittent latency and lost reachability from time to time.

The status page warned that users might see connection timeouts and errors for all services deployed in the Dallas data center, and might have trouble reaching Cloud Manager and the Linode API.

By 00:55 the responders found a large amount of broadcast traffic hitting the hypervisors, most of it from ARP requests that went unanswered and set off more broadcasts. Finding the broadcast storms did not end the instability: the team still had to find what was behind them and a way to stop them, and the network stayed unstable until about 04:20.

Symptom Intermittent connection failures High latency Intermittent timeouts

Root cause

Root cause DNS or network fault

Component Network switch

Contributing factor –

Trigger: –

Detection

Monitoring systems alerted on reachability problems in the Dallas data center at about 00:05 UTC, and an incident was raised at 00:10.

Remediation

Deployed a filter to the fleet that stemmed the broadcasts, so hosts regained CPU and answered ARP requests
Tested the temporary filter further and made it permanent

Action items

Find the root cause of the broadcast storms, taking the other recent Dallas incidents into account
Put processes in place to keep the infrastructure stable while upgrades are applied to the active network

Lessons

–

Sources

https://status.linode.com/incidents/n7yt1l21qtyz · official-status · archived copy

Related cases

–

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Network switch among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0562
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0562/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.