Incident Library

IL-0532 Microsoft Azure · 2022-06-28azurenetworking

Azure Software Load Balancer sent packets of one flow to different backends, dropping Firewall and Private Link traffic

Azure's Software Load Balancer (SLB) routes and load balances all network traffic within Azure datacenters and clusters. It is a set of servers called multiplexers, or MUXs, grouped into SLB scale units. Each scale unit hosts many VIPs and directs their traffic to the DIPs, the IPs of the resources behind each VIP, that health probes show to be in service. The order in which a MUX stores the DIPs decides which backend receives a given connection. A change in the latest SLB code deployment made some MUX instances store the DIPs in a different order, but only for load balancers with more than 15 DIPs in the backend pool. Traffic that crossed more than one MUX could be sent to a different backend at each MUX.

That happened in two ways. Azure Firewall is a stateful firewall that scales out behind an Azure Load Balancer and pins each flow to one instance, so when the incoming and return paths of a flow crossed different MUXs, its packets reached different firewall instances and were dropped. Private Link and service endpoint traffic also had a changed flow identifier in an IPv6 packet routing header, so when a connection moved from one MUX to another after a routing change in the physical network, the new MUX forwarded it to a different backend server, which rejected it. Telemetry showed a small rise in errors and customers opened support requests, but because the failures were intermittent, the health checks of Safe Deployment Practices did not tie these signals to the change, and the code reached 14 regions before the impact was correlated with it. Separately, an independent Azure WAN incident (YKDK-TT8) overlapped in time.

Engineers first mitigated the VIPs of customers identified as impacted by hand. In parallel engineers built a hotfix that could be applied without further impact to traffic, tested it and rolled it out in phases under Safe Deployment Practices, and ran automation that scanned all VIPs every five minutes and fixed any VIP in an inconsistent state until the hotfix was fully deployed. Consistent ordering of backend instances was restored to the state before the incident, and impact ended at 04:00 UTC on 1 July 2022.

Severity
–
Scope
Connections to resources behind the Software Load Balancer for Azure Firewall deployments with more than 15 backend instances and for service endpoint and Private Link connections to services with more than 15 DIPs, such as SQL and Storage
Users affected
A subset of Azure customers
Region
14 regions for the Azure Firewall scenario, three regions for the service endpoint and Private Link scenario
Time to detect
–
Time to resolve
70h 34m

Symptoms

From 05:26 UTC on 28 June 2022, some Azure customers saw intermittent network failures and packet loss when connecting to resources behind a Software Load Balancer (SLB). Telemetry showed a small increase in errors and customers raised support requests, but the failures were intermittent and were not linked to a recent deployment at first.

Two kinds of load balancer frontend IPs (VIPs) were hit. The first was Azure Firewall deployments with more than 15 destination IP addresses (DIPs) behind an Azure Load Balancer, where the upstream and downstream packets of one flow must land on the same DIP; packets of one flow reached different firewall instances and were dropped. This scenario affected services in 14 regions. The second was service endpoint or Private Link connections to a service in the region with more than 15 DIPs, for example SQL or Storage; when a connection moved from one MUX to another, the new MUX sent it to a different backend server, which rejected the request. This scenario affected services in three regions.

Correlating the customer reports with the platform took longer because the number of instances behind the internal load balancer of an Azure Firewall depends on each customer's traffic pattern, and not every firewall's load balancer had more than 15 instances. By the time customer impact was correlated with the deployment, the code had reached 14 of more than 60 regions. Impact continued until 04:00 UTC on 1 July.

Symptom Intermittent connection failuresPacket lossMisrouted requests

Root cause

Root cause Faulty deployment

Component Cloud load balancer
Contributing factor Ignored early signal
Trigger
A new SLB code deployment, rolled out through Safe Deployment Practices, made some MUX instances store the backend DIPs of load balancers with more than 15 DIPs in a different order.

Detection

Telemetry showed a small increase in errors and customers raised support requests, but the health checks of Safe Deployment Practices did not correlate these intermittent signals with the deployment until the code had reached 14 regions.

Remediation

Action items

Lessons

Microsoft wrote that the incident showed the need for more thorough end-to-end monitoring during safe deployments.

Sources

References

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Cloud load balancer among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.