Cases3 cases

1 to 3 of 3

1. IL-0499 Discord voice and video calls hung at Awaiting Endpoint for three hours after 17% of sessions stopped at once

   Discord2026

   Symptoms

   From 12:13 to 15:30 PDT on 25 March 2026, voice and video on Discord were heavily degraded. Users mostly could not start or join calls and saw "Awaiting Endpoint" in their call status.

   Service outage

   Trigger

   A routine change to tune the sessions service cluster on Kubernetes gave each pod more CPU and memory and lowered the pod count in proportion. When it was deployed to the first zone at 12:13 PDT, Kubernetes terminated 50% of that zone's pods, and the termination grace period ran out before the sessions could be handed off.

   Root cause

   Configuration change

   Component

   Pod terminationCoordination and key-value storeDatabase proxy and connection pooler

   Contributing factor

   Retry stormUnknown limit

   Remediation

   • Set low syncer spawn rate limits in the calls, streams and guilds services and restarted the voice syncers cluster
   • Restarted voice syncers instances one at a time, and four of the first five recovered fully
   • Brought 15 new voice syncers instances online, doubling capacity and halving the syncers per instance
   • Restarted the last backed-up voice syncers instance, leaving the doubled cluster fully healthy

   Action items

   • Add a validating admission webhook that rejects a scale-down of Elixir workloads until each pod has handed off its processes
   • Replace the Holster.Pool supervisor with a PartitionSupervisor so outgoing connections are created and checked out concurrently
   • Move gun connection process lifecycle management into Holster.Pool so the gun connection supervisor is no longer used
   • Tune the upstream rate limits that call into voice syncers, with load shedding tooling that can force them
   • Add rate limits on endpoint selection in voice syncers to limit outbound connection creation
   • Monitor the HTTP connection pooling mailboxes
   • Review and expand instrumentation and logging for service discovery announcement and voice syncer RPCs to SFUs
   • Monitor the most critical processes across all services instead of aggregate metrics such as overall mailbox health
   • Bring more workloads into Kubernetes so capacity can be scaled with a single command instead of provisioning VMs by hand
   • Re-architect the A/V system to scale its control plane services horizontally across multiple regions

   Lessons

   Discord says a large enough traffic spike will find a bottleneck somewhere; in earlier large-load incidents its ScyllaDB databases limited recovery, and this time the bottleneck moved to the voice services. Facing a capacity constraint, the team asks whether to increase supply, such as adding services, or reduce demand, such as rate limiting inbound traffic. During an incident it matters to know the toolbox, and Elixir and the BEAM give engineers many introspection options. Any slowdown in the hot loop of a single-threaded process can degrade both the service and the user experience.

   View case
2. IL-0461 Every rollout of Datadog's metrics query service on Kubernetes raised errors that the logs blamed on DNS

   Datadog2022

   Symptoms

   From September 2021, each rollout of an update to Datadog's metrics query service, which serves real-time and historical metric data to the web frontend and to monitor evaluation, came with a rise in errors. Clients retried failed queries, so few errors reached users, but the retries slowed dashboards and monitors. The service's logs showed DNS errors stopping it from connecting to the data stores it depends on.

   High error rateDNS resolution failuresHigh latencyOOMKilledPacket lossStale endpoint routing

   Trigger

   Rollouts of the metrics query service, which delete the old pods. The client behaviour behind the failures came from an August 2021 rollout of the metrics query clients that applied round_robin as the new default gRPC load balancing policy.

   Root cause

   Configuration change

   Component

   conntrack tableCiliumLinux kernelCoreDNSClient-side DNS resolver and cacheexternal-dnsHTTP and RPC client library

   Contributing factor

   Change with a delayed effectRetry stormUnknown limitMonitoring blind spot

   Detection

   Errors rose during rollouts of the metrics query service, and the service's logs showed DNS errors when connecting to its data stores.

   Remediation

   • Raised the memory of the node-local-dns pods from 64 MB to 256 MB, which stopped the OOM kills but not the DNS errors
   • Tested larger instance types and confirmed that scaling up the service's infrastructure resolved the errors
   • Removed the custom gRPC reconnect parameters from the clients to restore the defaults

   Action items

   • Contribute a change to Cilium that adds an unreachable route for a deleted pod's IP so clients get an ICMP error

   Lessons

   The errors looked like DNS but were not. ENA metrics and VPC Flow Logs gave the visibility into VPC connection tracking that host metrics did not, and Cilium, Kubernetes and AWS networking interact in ways that can make reverse path filtering drop packets it should allow. DNS propagation time depends on many settings. The causes were changes Datadog had made itself, to work around a dependency bug, to resolve earlier incidents and to override default reconnect parameters, which looked safe at the time and showed their effects much later, when the link was hard to see.

   View case
3. IL-0361 Job searches on seek.com.au failed for 16 minutes as a Transit Gateway silently dropped 9001-byte packets

   SEEK2021

   Symptoms

   In May 2021 some users of seek.com.au got errors when they searched for jobs, for 16 minutes. SEEK was in the middle of moving the traffic between two VPCs, the one that carried searches, from a VPC peering connection to an AWS Transit Gateway, and the team noticed quickly that search was broken.

   High error ratePacket lossIntermittent timeouts

   Trigger

   In the second step of the migration, SEEK changed the routing table of the first VPC to use the transit gateway while the second VPC still routed over the peering connection, so the two directions of each connection took different paths.

   Root cause

   Configuration change

   Component

   Transit gateway or network interconnect

   Contributing factor

   Unknown limitUnreviewed defaultsMonitoring blind spot

   Detection

   The team noticed soon after the routing change that search on seek.com.au was not working.

   Remediation

   • Rolled back the migration of the search traffic's peering connection to the transit gateway

   Action items

   • Lower the MTU of EC2 instances to 8500, on Amazon Linux 2 with a script at /sbin/ifup-local that runs whenever a network interface comes up

   Lessons

   A transit gateway has a smaller MTU than a same-region peering connection, drops oversized packets silently and clamps the MSS only on packets that cross it. When a migration leaves the two directions of a TCP connection on different paths, the clamp lands on the wrong side of the handshake and large packets are lost; the fix is not to send packets larger than 8500 bytes.

   View case

By root cause1