Cases3 cases

1 to 3 of 3

1. IL-0461 Every rollout of Datadog's metrics query service on Kubernetes raised errors that the logs blamed on DNS

   Datadog2022

   Symptoms

   From September 2021, each rollout of an update to Datadog's metrics query service, which serves real-time and historical metric data to the web frontend and to monitor evaluation, came with a rise in errors. Clients retried failed queries, so few errors reached users, but the retries slowed dashboards and monitors. The service's logs showed DNS errors stopping it from connecting to the data stores it depends on.

   High error rateDNS resolution failuresHigh latencyOOMKilledPacket lossStale endpoint routing

   Trigger

   Rollouts of the metrics query service, which delete the old pods. The client behaviour behind the failures came from an August 2021 rollout of the metrics query clients that applied round_robin as the new default gRPC load balancing policy.

   Root cause

   Configuration change

   Component

   conntrack tableCiliumLinux kernelCoreDNSClient-side DNS resolver and cacheexternal-dnsHTTP and RPC client library

   Contributing factor

   Change with a delayed effectRetry stormUnknown limitMonitoring blind spot

   Detection

   Errors rose during rollouts of the metrics query service, and the service's logs showed DNS errors when connecting to its data stores.

   Remediation

   • Raised the memory of the node-local-dns pods from 64 MB to 256 MB, which stopped the OOM kills but not the DNS errors
   • Tested larger instance types and confirmed that scaling up the service's infrastructure resolved the errors
   • Removed the custom gRPC reconnect parameters from the clients to restore the defaults

   Action items

   • Contribute a change to Cilium that adds an unreachable route for a deleted pod's IP so clients get an ICMP error

   Lessons

   The errors looked like DNS but were not. ENA metrics and VPC Flow Logs gave the visibility into VPC connection tracking that host metrics did not, and Cilium, Kubernetes and AWS networking interact in ways that can make reverse path filtering drop packets it should allow. DNS propagation time depends on many settings. The causes were changes Datadog had made itself, to work around a dependency bug, to resolve earlier incidents and to override default reconnect parameters, which looked safe at the time and showed their effects much later, when the link was hard to see.

   View case
2. IL-0252 Spotify users who were logged out could not log back in during a global outage of service discovery

   Spotify2022

   Symptoms

   At 18:12 UTC on 8 March 2022, reports began to surface of users being logged out of the different Spotify client apps. Anyone who was logged out of a Spotify app could not log back in, and the outage was global.

   High error rate

   Trigger

   Google Cloud Traffic Director, the xDS-based traffic control plane and discovery system that some Spotify services use, had an outage.

   Root cause

   Cloud provider outage

   Component

   Service discoveryHTTP and RPC client library

   Contributing factor

   Overlapping failures

   Detection

   Reports of users being logged out of the client apps started to surface at 18:12 UTC.

   Remediation

   • Rolled out configuration changes that moved the affected systems back to the DNS-based service discovery

   Action items

   • Work with Google Cloud to understand how the Traffic Director issues led to a large outage for Spotify users
   • Add monitoring and alerting that catches similar service discovery problems earlier
   • Add safety nets for monitoring, automatic error detection and self-recovery

   View case
3. IL-0253 Firefox froze and could not load any web content for close to two hours, stuck in an HTTP/3 request to Telemetry

   Mozilla2022

   Symptoms

   On 13 January 2022, shortly after 07:28 UTC, Mozilla saw a spike in crashes reported through the Firefox crash reporter and received several reports, from inside and outside Mozilla, of the browser hanging. For close to two hours Firefox was unusable for users worldwide and could not load web content.

   Service outage

   Trigger

   At 07:28 UTC on 13 January 2022 GCP deployed an unannounced change that made HTTP/3 the default for load balancers set to "Automatic (default)", so some Firefox connections to Mozilla's services, including Telemetry, started using HTTP/3 instead of HTTP/2.

   Root cause

   Software defect

   Component

   Cloud load balancerHTTP and RPC client library

   Contributing factor

   Unreviewed defaultsGap in automated tests

   Detection

   A spike in crashes reported through the Firefox crash reporter, and reports of browser hangs from inside and outside Mozilla.

   Remediation

   • Explicitly disabled HTTP/3 on the GCP load balancers at 09:12 UTC

   Action items

   • Work with GCP to improve the situation around unannounced changes such as the HTTP/3 default
   • Review all service configurations to avoid similar mistakes
   • Run more system tests with different HTTP versions
   • Find ways to make the browser more resilient to such problems and to speed up incident response

   Lessons

   The hang needed three things together: a provider-side change to the HTTP version, a Rust component calling the network stack through viaduct, and a case-sensitive header lookup deep in the HTTP/3 code. Leaving the load balancers on an automatic default let the provider's change apply without any action from Mozilla, and an announcement would likely have led to a controlled test in staging first. A change of HTTP version and an extra networking layer such as viaduct are significant enough to be covered by system tests.

   View case

By root cause3