Cases24 cases

14 to 18 of 24

1. IL-0225 An IGP purge command made every Microsoft WAN router recompute routes, causing global latency, timeouts and packet loss

   Microsoft Azure2023

   Symptoms

   From 07:08 UTC on 25 January 2023, customers saw long network latency and timeouts when connecting to resources in Azure regions and to other Microsoft services such as Microsoft 365 and Power Platform. Microsoft's monitoring detected DNS and WAN issues from 07:11 UTC.

   High latencyIntermittent timeoutsIntermittent connection failuresPacket lossNetwork unreachable

   Trigger

   A network engineer adding capacity to the global WAN in Madrid ran a step of a recently changed standard operating procedure that purged the IGP database; on the router manufacturer involved, that command runs on all IGP-joined routers instead of only the local one.

   Root cause

   Configuration change

   Component

   Backbone network (WAN)Interior routing (IGP)BGP routing

   Contributing factor

   No configuration testsUnverified assumptionExcessive blast radiusHarmful manual action

   Detection

   Monitoring detected DNS and WAN issues from 07:11 UTC and alerted the on-call engineers within 5 minutes of the command being run. The team reviewed all recent changes and identified the command by 08:20 UTC. The engineer making the change was not informed of the alerts because of the unqualified changes to the SOP.

   Remediation

   • Restarted by hand the paused WAN health and traffic engineering systems, which ended the remaining packet loss

   Action items

   • Audit and block similar widespread-impact commands across all three vendors for all WAN router roles
   • Give on-call engineers real-time visibility of approved and unqualified activity on network devices
   • Run regular mandatory operational training and attestation of following all SOPs
   • Have the Change Advisory Board review all SOPs still pending qualification within 30 days

   View case
2. IL-0532 Azure Software Load Balancer sent packets of one flow to different backends, dropping Firewall and Private Link traffic

   Microsoft Azure2022

   Symptoms

   From 05:26 UTC on 28 June 2022, some Azure customers saw intermittent network failures and packet loss when connecting to resources behind a Software Load Balancer (SLB). Telemetry showed a small increase in errors and customers raised support requests, but the failures were intermittent and were not linked to a recent deployment at first.

   Intermittent connection failuresPacket lossMisrouted requests

   Trigger

   A new SLB code deployment, rolled out through Safe Deployment Practices, made some MUX instances store the backend DIPs of load balancers with more than 15 DIPs in a different order.

   Root cause

   Faulty deployment

   Component

   Cloud load balancer

   Contributing factor

   Ignored early signal

   Detection

   Telemetry showed a small increase in errors and customers raised support requests, but the health checks of Safe Deployment Practices did not correlate these intermittent signals with the deployment until the code had reached 14 regions.

   Remediation

   • Mitigated the VIPs of customers identified as impacted with targeted manual operations
   • Built and tested a hotfix and rolled it out in phases under Safe Deployment Practices
   • Ran automation that scanned all VIPs every five minutes and fixed VIPs in an inconsistent state until the hotfix was out
   • Restored consistent ordering of the backend instances to the state before the incident

   Action items

   • Add Azure Firewall monitoring that detects when the upstream and downstream packets of one flow do not reach the same DIP
   • Keep the IPv6 flow identifier consistent for the lifetime of a connection so all its packets land on one MUX
   • Improve packet capture support for firewalls
   • Add logging of TCP flows, including invalid packets and RST, FIN and FIN-ACK packets
   • Onboard more Azure services to the Gandalf AIOps solution suite for end-to-end monitoring during safe deployments

   Lessons

   Microsoft wrote that the incident showed the need for more thorough end-to-end monitoring during safe deployments.

   View case
3. IL-0461 Every rollout of Datadog's metrics query service on Kubernetes raised errors that the logs blamed on DNS

   Datadog2022

   Symptoms

   From September 2021, each rollout of an update to Datadog's metrics query service, which serves real-time and historical metric data to the web frontend and to monitor evaluation, came with a rise in errors. Clients retried failed queries, so few errors reached users, but the retries slowed dashboards and monitors. The service's logs showed DNS errors stopping it from connecting to the data stores it depends on.

   High error rateDNS resolution failuresHigh latencyOOMKilledPacket lossStale endpoint routing

   Trigger

   Rollouts of the metrics query service, which delete the old pods. The client behaviour behind the failures came from an August 2021 rollout of the metrics query clients that applied round_robin as the new default gRPC load balancing policy.

   Root cause

   Configuration change

   Component

   conntrack tableCiliumLinux kernelCoreDNSClient-side DNS resolver and cacheexternal-dnsHTTP and RPC client library

   Contributing factor

   Change with a delayed effectRetry stormUnknown limitMonitoring blind spot

   Detection

   Errors rose during rollouts of the metrics query service, and the service's logs showed DNS errors when connecting to its data stores.

   Remediation

   • Raised the memory of the node-local-dns pods from 64 MB to 256 MB, which stopped the OOM kills but not the DNS errors
   • Tested larger instance types and confirmed that scaling up the service's infrastructure resolved the errors
   • Removed the custom gRPC reconnect parameters from the clients to restore the defaults

   Action items

   • Contribute a change to Cilium that adds an unreachable route for a deleted pod's IP so clients get an ICMP error

   Lessons

   The errors looked like DNS but were not. ENA metrics and VPC Flow Logs gave the visibility into VPC connection tracking that host metrics did not, and Cilium, Kubernetes and AWS networking interact in ways that can make reverse path filtering drop packets it should allow. DNS propagation time depends on many settings. The causes were changes Datadog had made itself, to work around a dependency bug, to resolve earlier incidents and to override default reconnect parameters, which looked safe at the time and showed their effects much later, when the link was hard to see.

   View case
4. IL-0361 Job searches on seek.com.au failed for 16 minutes as a Transit Gateway silently dropped 9001-byte packets

   SEEK2021

   Symptoms

   In May 2021 some users of seek.com.au got errors when they searched for jobs, for 16 minutes. SEEK was in the middle of moving the traffic between two VPCs, the one that carried searches, from a VPC peering connection to an AWS Transit Gateway, and the team noticed quickly that search was broken.

   High error ratePacket lossIntermittent timeouts

   Trigger

   In the second step of the migration, SEEK changed the routing table of the first VPC to use the transit gateway while the second VPC still routed over the peering connection, so the two directions of each connection took different paths.

   Root cause

   Configuration change

   Component

   Transit gateway or network interconnect

   Contributing factor

   Unknown limitUnreviewed defaultsMonitoring blind spot

   Detection

   The team noticed soon after the routing change that search on seek.com.au was not working.

   Remediation

   • Rolled back the migration of the search traffic's peering connection to the transit gateway

   Action items

   • Lower the MTU of EC2 instances to 8500, on Amazon Linux 2 with a script at /sbin/ifup-local that runs whenever a network interface comes up

   Lessons

   A transit gateway has a smaller MTU than a same-region peering connection, drops oversized packets silently and clamps the MSS only on packets that cross it. When a migration leaves the two directions of a TCP connection on different paths, the clamp lands on the wrong side of the handshake and large packets are lost; the fix is not to send packets larger than 8500 bytes.

   View case
5. IL-0336 AWS Direct Connect traffic into the Tokyo Region had packet loss and intermittent connectivity for about six hours

   Amazon Web Services2021

   Symptoms

   From 7:30 AM JST on 2 September 2021, AWS Direct Connect customers saw intermittent connectivity problems and elevated packet loss on traffic going to their VPCs in the Tokyo (AP-NORTHEAST-1) Region. Internal AWS alarms on Direct Connect packet loss fired at the same time.

   Packet lossIntermittent connection failures

   Trigger

   Customer traffic consistently generated packets with a very specific set of attributes and contents, and these packets set off a latent defect in the network device operating system on one layer of the Direct Connect network in Tokyo.

   Root cause

   Software defect

   Component

   Transit gateway or network interconnect

   Contributing factor

   Change with a delayed effectUnrepresentative staging

   Detection

   Internal alarms alerted AWS engineers to elevated packet loss for Direct Connect customers connecting to the Tokyo Region at 7:30 AM JST. The automation that monitors network devices also saw a higher rate of failed devices than normal and alerted engineers.

   Remediation

   • Removed the failing network devices from service so healthy devices carried the traffic, which helped only for a time
   • Reset failed devices and brought them back into service slowly, without stopping the failures
   • Disabled the new protocol in a single Availability Zone, then across the Tokyo Region

   Action items

   • Apply the change carefully to all other AWS Regions, with a new way to detect and remediate the issue before customers are affected

   View case

By root cause9