GitHub was slow and erroring all day and down for 18 minutes as links between access and aggregation switches saturated

Symptoms

On Friday, 30 November 2012, GitHub had sporadic bursts of slow responses and intermittent errors for the whole day and was completely unavailable for 18 minutes. One fileserver pair was offline for a large part of the day, which affected a small percentage of repositories; this was a side effect of the network problems on the high-availability clustering between the fileserver nodes.

The problems began during a planned migration of the datacenter switches to a new aggregation network. When the final A switch was connected, GitHub lost connectivity with the B switch in the same cabinet: a misconfiguration on that pair of switches had caused a "bridge loop", and the switches disabled links to protect the network. After that was fixed and the migration completed, performance stayed below expectations, and all of the connections between the access switches and the aggregation switches were completely saturated.

Engineers diagnosed the saturation as a "broadcast storm" from a bridge loop that had gone undetected, and spent most of the day auditing switch configurations port by port. While disconnecting single access-to-aggregation links to narrow the problem down, they found that taking down one link of a redundant pair made the access switch disable its redundant link too, so the network could not survive the loss of an aggregation switch. With the switch vendor they found the cause: a setting meant to detect partial link failure is not supported between the aggregation and access switch models, and shutting down one link made it falsely disable all the links. The 18 minutes of hard downtime happened during this troubleshooting, when connectivity to several switches was lost at the same time. After that setting was removed, a couple more hours of testing links found no loop.

The cause turned out to be different. In a five-hour investigation with the switch vendor on Friday night, GitHub found that the aggregation switches were missing many MAC addresses from their MAC address tables and could not learn a significant share of them, so they flooded traffic for those devices to every port. That flooded traffic was enough to saturate all of the links between the access and aggregation switches.

Symptom High latencyHigh error rateService outage

Root cause

Trigger

On the morning of 30 November GitHub moved the A switches in its server cabinets onto the new aggregation switches, the second half of a migration whose B-switch half had been done the day before.

Detection

Engineers saw the problems during the maintenance itself: connectivity to the B switch was lost as the last A switch was connected, and afterwards performance stayed low and the links between access and aggregation switches showed as completely saturated.

Remediation

• Fixed the bridge-loop problem and returned the affected B switch to service
• Removed the partial link failure detection setting from the access switches
• Escalated to the network vendor, which raised the case to its highest severity level
• Restarted core processes on the aggregation switches in a Saturday 0600 Pacific maintenance window so they learned MAC addresses again

Action items

• Deploy the vendor's software update that fixes the MAC learning issue to the switches
• Closely monitor aggregation-to-access capacity and keep a workaround process ready if the problem returns
• Build a duplicate of the network stack, from the routers through the access switches, as a staging environment for testing such changes
• Add automated network monitoring that alerts sooner on similar issues
• Run scheduled incident response exercises with scenarios that counter tunnel vision
• Contact the network vendor's support at the first sign of trouble in the network

Lessons

GitHub had designed the maintenance to have no impact on customers and clearly failed, so it plans to invest in a staging copy of its network to test such changes and catch bugs like this one. During the response the team fixated for a long time on a bridge loop, which kept it from considering other causes. GitHub had a very positive experience with the vendor's support staff, so it will contact that support team at the first sign of network trouble.

Sources

• https://github.blog/news-insights/the-library/network-problems-last-friday/ · blog · archived copy

Related cases

• IL-0481 RoCEv2 GIDs changed after RNIC port flaps, collapsing the communication collective and crashing LLM training jobsInfrawaves · 2026

  same root cause · shared NIC, cable and switch port link

• IL-0347 About 25% of Scaleway FR-PAR-1 hypervisors lost VPC connectivity when both VPC BGP route reflectors got stuckScaleway · 2024

  same root cause · shared Network switch

• IL-0344 Linode's US-West data centers in Fremont and Santa Clara lost connectivity as the fibre circuits between them failedLinode · 2023

  same root cause · shared NIC, cable and switch port link

Check your cluster for this

For AI tools and scripts

• This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0263
• The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0263/similar
• MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.