Cases5 cases

1 to 5 of 5

1. IL-0257 canva.com went down for about 50 minutes when 1.5 million requests per second OOM-killed every API Gateway task

   Canva2024

   Symptoms

   On 12 November 2024, after a new version of Canva's editor was deployed at 8:47 AM UTC, the editor's object panel stayed in a perpetual loading state for many users, particularly in Asia. Then from 9:08 AM UTC to about 10:00 AM UTC canva.com was unavailable and all requests to it failed.

   Service outageOOMKilledHigh latencyPacket loss

   Trigger

   At 8:47 AM UTC a new version of the Canva editor page was deployed, and client devices began fetching its new static assets through Cloudflare while the network path between Cloudflare's Singapore and Ashburn locations was losing packets.

   Root cause

   Dependency failure

   Component

   CDN and edge networkAPI gatewayAmazon ECS

   Contributing factor

   Overlapping failuresMonitoring blind spotRetry stormAutomation amplificationUnpatched known bug

   Remediation

   • Added a temporary Cloudflare firewall rule at 9:29 AM UTC that blocked all traffic at the CDN so new API Gateway tasks could start
   • Redirected canva.com to the status page
   • Restored traffic gradually, starting with Australian users under strict rate limits

   Action items

   • Write a runbook for rerouting, blocking and progressively restoring traffic
   • Show users a more informative error page when canva.com is unavailable
   • Raise the baseline number of API Gateway tasks and the memory of each task
   • Add load shedding rules to the API Gateway for similar traffic patterns
   • Load test the API Gateway regularly
   • Deploy the patch for the thread-locking bug in the telemetry library
   • Add multithreaded tests for the contended code path to the telemetry library microbenchmark suite
   • Use page load completion events as a second canary indicator for page asset releases
   • Experiment with adding another rollout stage to the canary to give more time to detect issues
   • Add timeouts so user requests do not wait excessively long for assets
   • Work with Cloudflare to understand the system interactions behind the incident

   View case
2. IL-0339 Azure Front Door and Azure CDN users saw intermittent connection errors, timeouts and latency spikes for hours

   Microsoft Azure2024

   Symptoms

   From 11:45 UTC on 30 July 2024, a subset of customers got intermittent connection errors, timeouts and latency spikes when connecting to Microsoft services that use Azure Front Door (AFD) and Azure Content Delivery Network (CDN). The Azure Portal team saw the first service degradation at 11:47 UTC.

   Intermittent connection failuresIntermittent timeoutsHigh latencyPacket loss

   Trigger

   Around 11:45 UTC the Azure network DDoS protection service was disengaging after a routine DDoS mitigation and returning traffic to its default routing to AFD, but the routes at one site in Europe could not be updated because the network control plane could not reach that site during a local power outage.

   Root cause

   Configuration change

   Component

   CDN and edge networkDDoS protection serviceNetwork control plane (SDN)Data center power and cooling

   Contributing factor

   Overlapping failuresExcessive blast radiusMonitoring blind spot

   Detection

   The Azure Portal team detected service degradation at 11:47 UTC and began to investigate. At 12:10 UTC network monitoring correlated the Portal incident to a network issue at one site in Europe, and networking engineers joined.

   Remediation

   • Rerouted traffic under the standard congestion playbook, which removed most of the impact by 13:58 UTC
   • Ordered the network control plane to reissue its commands to the misbehaving device, which could not be reached
   • Removed the device that kept attracting traffic from the network and scanned the network for other issues

   Action items

   • Add the missing configuration on network devices so a DDoS mitigation issue in one geography cannot spread to another
   • Improve validation and monitoring in the Azure network to detect invalid configurations
   • Improve monitoring for when the DDoS protection service is unreachable from the control plane but still serving traffic

   View case
3. IL-0540 Korea's local government administrative services and Government24 went down as a router port dropped most large packets

   Korea Ministry of the Interior and Safety2023

   Symptoms

   On Friday 17 November 2023, the local government administrative computer services in Korea stopped working, and the Government24 service of the National Information Resources Service was down until the next morning.

   Service outagePacket lossHigh latency

   Root cause

   Hardware failure

   Component

   NIC, cable and switch port link

   Contributing factor

   Monitoring blind spotGray failure

   Remediation

   • Replaced the two malfunctioning L4 devices with higher-performance devices, tested them and resumed Government24 at 09:00 on 18 November
   • Moved the connection on the defective port of the Daejeon center router to another port at 07:00 on 19 November
   • Swapped out the module of the router that had the fault

   Action items

   • Inspect all old equipment that may have similar port defects
   • Revise the incident handling manual so the public is told about an outage quickly
   • Announce the status of service recovery quickly through various means
   • Build integrated monitoring of critical service systems and related equipment to catch signs of failure early
   • Form a standing incident response team to send into major outages
   • Prepare administrative measures and a response manual so administrative services can still be provided when core digital government services stop
   • Name a paralysis of the national computer network as a type of disaster and accident, managed from prevention to recovery
   • Build a government-wide crisis response system for linked information systems and strengthen the control tower for outage prevention and response
   • Improve how public IT projects are run and managed, replace hardware and software past their service life quickly and make project pricing realistic
   • Reduce the reliance on outsourcing for developing and operating information systems and bring digital talent into government
   • Review the redundancy, recovery systems and network configuration of the systems in the National Information Resources Service, and its organisation
   • Work with manufacturers on a way to find and monitor failures of internal parts earlier

   Lessons

   The team said that hardware faults like this defective port also happen in the private sector, and that failures of parts inside a device are hard to catch in advance with basic monitoring and daily visual checks. Redundancy did not help because it takes over only when a whole device stops working, and a module that failed only in part did not make the device count as abnormal. The impact was wide because the defective port carried traffic for all the services.

   View case
4. IL-0461 Every rollout of Datadog's metrics query service on Kubernetes raised errors that the logs blamed on DNS

   Datadog2022

   Symptoms

   From September 2021, each rollout of an update to Datadog's metrics query service, which serves real-time and historical metric data to the web frontend and to monitor evaluation, came with a rise in errors. Clients retried failed queries, so few errors reached users, but the retries slowed dashboards and monitors. The service's logs showed DNS errors stopping it from connecting to the data stores it depends on.

   High error rateDNS resolution failuresHigh latencyOOMKilledPacket lossStale endpoint routing

   Trigger

   Rollouts of the metrics query service, which delete the old pods. The client behaviour behind the failures came from an August 2021 rollout of the metrics query clients that applied round_robin as the new default gRPC load balancing policy.

   Root cause

   Configuration change

   Component

   conntrack tableCiliumLinux kernelCoreDNSClient-side DNS resolver and cacheexternal-dnsHTTP and RPC client library

   Contributing factor

   Change with a delayed effectRetry stormUnknown limitMonitoring blind spot

   Detection

   Errors rose during rollouts of the metrics query service, and the service's logs showed DNS errors when connecting to its data stores.

   Remediation

   • Raised the memory of the node-local-dns pods from 64 MB to 256 MB, which stopped the OOM kills but not the DNS errors
   • Tested larger instance types and confirmed that scaling up the service's infrastructure resolved the errors
   • Removed the custom gRPC reconnect parameters from the clients to restore the defaults

   Action items

   • Contribute a change to Cilium that adds an unreachable route for a deleted pod's IP so clients get an ICMP error

   Lessons

   The errors looked like DNS but were not. ENA metrics and VPC Flow Logs gave the visibility into VPC connection tracking that host metrics did not, and Cilium, Kubernetes and AWS networking interact in ways that can make reverse path filtering drop packets it should allow. DNS propagation time depends on many settings. The causes were changes Datadog had made itself, to work around a dependency bug, to resolve earlier incidents and to override default reconnect parameters, which looked safe at the time and showed their effects much later, when the link was hard to see.

   View case
5. IL-0361 Job searches on seek.com.au failed for 16 minutes as a Transit Gateway silently dropped 9001-byte packets

   SEEK2021

   Symptoms

   In May 2021 some users of seek.com.au got errors when they searched for jobs, for 16 minutes. SEEK was in the middle of moving the traffic between two VPCs, the one that carried searches, from a VPC peering connection to an AWS Transit Gateway, and the team noticed quickly that search was broken.

   High error ratePacket lossIntermittent timeouts

   Trigger

   In the second step of the migration, SEEK changed the routing table of the first VPC to use the transit gateway while the second VPC still routed over the peering connection, so the two directions of each connection took different paths.

   Root cause

   Configuration change

   Component

   Transit gateway or network interconnect

   Contributing factor

   Unknown limitUnreviewed defaultsMonitoring blind spot

   Detection

   The team noticed soon after the routing change that search on seek.com.au was not working.

   Remediation

   • Rolled back the migration of the search traffic's peering connection to the transit gateway

   Action items

   • Lower the MTU of EC2 instances to 8500, on Amazon Linux 2 with a script at /sbin/ifup-local that runs whenever a network interface comes up

   Lessons

   A transit gateway has a smaller MTU than a same-region peering connection, drops oversized packets silently and clamps the MSS only on packets that cross it. When a migration leaves the two directions of a TCP connection on different paths, the clamp lands on the wrong side of the handshake and large packets are lost; the fix is not to send packets larger than 8500 bytes.

   View case

By root cause3