Incident Library
Applied filters Root causeFaulty deploymentSymptomMisrouted requestsFactorIgnored early signalClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0532 Azure Software Load Balancer sent packets of one flow to different backends, dropping Firewall and Private Link traffic
    Microsoft Azure2022
    Symptoms

    From 05:26 UTC on 28 June 2022, some Azure customers saw intermittent network failures and packet loss when connecting to resources behind a Software Load Balancer (SLB). Telemetry showed a small increase in errors and customers raised support requests, but the failures were intermittent and were not linked to a recent deployment at first.

    Intermittent connection failuresPacket lossMisrouted requests
    Trigger

    A new SLB code deployment, rolled out through Safe Deployment Practices, made some MUX instances store the backend DIPs of load balancers with more than 15 DIPs in a different order.

    Root cause
    Faulty deployment
    Component
    Cloud load balancer
    Contributing factor
    Ignored early signal
    Detection

    Telemetry showed a small increase in errors and customers raised support requests, but the health checks of Safe Deployment Practices did not correlate these intermittent signals with the deployment until the code had reached 14 regions.

    Remediation
    • Mitigated the VIPs of customers identified as impacted with targeted manual operations
    • Built and tested a hotfix and rolled it out in phases under Safe Deployment Practices
    • Ran automation that scanned all VIPs every five minutes and fixed VIPs in an inconsistent state until the hotfix was out
    • Restored consistent ordering of the backend instances to the state before the incident
    Action items
    • Add Azure Firewall monitoring that detects when the upstream and downstream packets of one flow do not reach the same DIP
    • Keep the IPv6 flow identifier consistent for the lifetime of a connection so all its packets land on one MUX
    • Improve packet capture support for firewalls
    • Add logging of TCP flows, including invalid packets and RST, FIN and FIN-ACK packets
    • Onboard more Azure services to the Gandalf AIOps solution suite for end-to-end monitoring during safe deployments
    Lessons

    Microsoft wrote that the incident showed the need for more thorough end-to-end monitoring during safe deployments.

    View case
Check your cluster for this

This case involves Cloud load balancer. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Faulty deployment1 case

Component
Cloud load balancer
Contributing factor
Ignored early signal
Organisation
Microsoft Azure
  1. IL-0532 Azure Software Load Balancer sent packets of one flow to different backends, dropping Firewall and Private Link traffic
    Microsoft Azure2022