Cases24 cases

16 to 20 of 24

1. IL-0461 Every rollout of Datadog's metrics query service on Kubernetes raised errors that the logs blamed on DNS

   Datadog2022

   Symptoms

   From September 2021, each rollout of an update to Datadog's metrics query service, which serves real-time and historical metric data to the web frontend and to monitor evaluation, came with a rise in errors. Clients retried failed queries, so few errors reached users, but the retries slowed dashboards and monitors. The service's logs showed DNS errors stopping it from connecting to the data stores it depends on.

   High error rateDNS resolution failuresHigh latencyOOMKilledPacket lossStale endpoint routing

   Trigger

   Rollouts of the metrics query service, which delete the old pods. The client behaviour behind the failures came from an August 2021 rollout of the metrics query clients that applied round_robin as the new default gRPC load balancing policy.

   Root cause

   Configuration change

   Component

   conntrack tableCiliumLinux kernelCoreDNSClient-side DNS resolver and cacheexternal-dnsHTTP and RPC client library

   Contributing factor

   Change with a delayed effectRetry stormUnknown limitMonitoring blind spot

   Detection

   Errors rose during rollouts of the metrics query service, and the service's logs showed DNS errors when connecting to its data stores.

   Remediation

   • Raised the memory of the node-local-dns pods from 64 MB to 256 MB, which stopped the OOM kills but not the DNS errors
   • Tested larger instance types and confirmed that scaling up the service's infrastructure resolved the errors
   • Removed the custom gRPC reconnect parameters from the clients to restore the defaults

   Action items

   • Contribute a change to Cilium that adds an unreachable route for a deleted pod's IP so clients get an ICMP error

   Lessons

   The errors looked like DNS but were not. ENA metrics and VPC Flow Logs gave the visibility into VPC connection tracking that host metrics did not, and Cilium, Kubernetes and AWS networking interact in ways that can make reverse path filtering drop packets it should allow. DNS propagation time depends on many settings. The causes were changes Datadog had made itself, to work around a dependency bug, to resolve earlier incidents and to override default reconnect parameters, which looked safe at the time and showed their effects much later, when the link was hard to see.

   View case
2. IL-0361 Job searches on seek.com.au failed for 16 minutes as a Transit Gateway silently dropped 9001-byte packets

   SEEK2021

   Symptoms

   In May 2021 some users of seek.com.au got errors when they searched for jobs, for 16 minutes. SEEK was in the middle of moving the traffic between two VPCs, the one that carried searches, from a VPC peering connection to an AWS Transit Gateway, and the team noticed quickly that search was broken.

   High error ratePacket lossIntermittent timeouts

   Trigger

   In the second step of the migration, SEEK changed the routing table of the first VPC to use the transit gateway while the second VPC still routed over the peering connection, so the two directions of each connection took different paths.

   Root cause

   Configuration change

   Component

   Transit gateway or network interconnect

   Contributing factor

   Unknown limitUnreviewed defaultsMonitoring blind spot

   Detection

   The team noticed soon after the routing change that search on seek.com.au was not working.

   Remediation

   • Rolled back the migration of the search traffic's peering connection to the transit gateway

   Action items

   • Lower the MTU of EC2 instances to 8500, on Amazon Linux 2 with a script at /sbin/ifup-local that runs whenever a network interface comes up

   Lessons

   A transit gateway has a smaller MTU than a same-region peering connection, drops oversized packets silently and clamps the MSS only on packets that cross it. When a migration leaves the two directions of a TCP connection on different paths, the clamp lands on the wrong side of the handshake and large packets are lost; the fix is not to send packets larger than 8500 bytes.

   View case
3. IL-0336 AWS Direct Connect traffic into the Tokyo Region had packet loss and intermittent connectivity for about six hours

   Amazon Web Services2021

   Symptoms

   From 7:30 AM JST on 2 September 2021, AWS Direct Connect customers saw intermittent connectivity problems and elevated packet loss on traffic going to their VPCs in the Tokyo (AP-NORTHEAST-1) Region. Internal AWS alarms on Direct Connect packet loss fired at the same time.

   Packet lossIntermittent connection failures

   Trigger

   Customer traffic consistently generated packets with a very specific set of attributes and contents, and these packets set off a latent defect in the network device operating system on one layer of the Direct Connect network in Tokyo.

   Root cause

   Software defect

   Component

   Transit gateway or network interconnect

   Contributing factor

   Change with a delayed effectUnrepresentative staging

   Detection

   Internal alarms alerted AWS engineers to elevated packet loss for Direct Connect customers connecting to the Tokyo Region at 7:30 AM JST. The automation that monitors network devices also saw a higher rate of failed devices than normal and alerted engineers.

   Remediation

   • Removed the failing network devices from service so healthy devices carried the traffic, which helped only for a time
   • Reset failed devices and brought them back into service slowly, without stopping the failures
   • Disabled the new protocol in a single Availability Zone, then across the Tokyo Region

   Action items

   • Apply the change carefully to all other AWS Regions, with a new way to detect and remediate the issue before customers are affected

   View case
4. IL-0220 An overloaded AWS Transit Gateway dropped packets on 4 January 2021, and Slack's web tier saturated and went down

   Slack2021

   Symptoms

   During the Americas' morning of 4 January 2021, an external monitoring service paged Slack because error rates were creeping up. While the investigation was starting, Slack's own dashboarding and alerting service became unavailable, so responders had to work without their usual dashboards and alerts.

   High error rateMonitoring outagePacket lossService outageScale-up failureFile descriptor exhaustionCloud API rate limit exhausted

   Trigger

   On the first working day after the holidays, Slack's traffic went from the quietest time of the year to one of its biggest days overnight, with cold client caches pulling more data than usual, and one of Slack's AWS Transit Gateways did not scale fast enough for it.

   Root cause

   Cloud provider outage

   Component

   Transit gateway or network interconnectAutomated capacity manager

   Contributing factor

   Circular dependencyAutomation amplification

   Detection

   An external monitoring service paged Slack when error rates started creeping up during the Americas' morning. AWS engineers were separately alerted to the packet drops by AWS's internal monitoring.

   Remediation

   • Disabled automated downscaling of the web tier to keep serving capacity and allow investigation
   • Cleared up the broken, partly provisioned instances that filled the autoscaling groups
   • Repaired provision-service so new web tier instances could be provisioned and enter service
   • AWS engineers increased the Transit Gateway capacity manually across all Availability Zones

   Action items

   • AWS reviews its Transit Gateway scaling algorithms for large packet-per-second increases
   • Request a preemptive upscaling of the Transit Gateways at the end of the next holiday season
   • Run the dashboard service instances in the same VPC as their database to remove the Transit Gateway dependency
   • Load test provision-service regularly
   • Reevaluate health-checking and autoscaling configurations so they cannot overload provision-service again

   Lessons

   Monitoring has to stay as independent as possible of the infrastructure it watches, or it fails when it is needed most. Here the dashboard and alerting services failed because they ran in a different VPC from their databases and so depended on the Transit Gateways. provision-service had been load tested before, but this event exceeded the earlier scale-ups.

   View case
5. IL-0240 Cloudflare API requests failed and the dashboard slowed for hours after the authentication database lost its replicas

   Cloudflare2020

   Symptoms

   On 2 November 2020 the Cloudflare API and dashboard were degraded for six hours and 33 minutes. The success rate of API queries dropped at times to 75%, and the dashboard was up to 80 times slower than normal, while Cloudflare's edge network kept working.

   High error rateHigh latencyDatabase overloadWrites rejectedPacket loss

   Trigger

   At 14:43 UTC a network switch in a control plane data center partly failed: it kept LACP and BGP running but stopped forwarding some packets, and it recovered on its own six minutes later.

   Root cause

   DNS or network fault

   Component

   NIC, cable and switch port linkCoordination and key-value storeDatabase failover managerDatabaseDatabase replication

   Contributing factor

   Automation amplificationNo headroomHarmful manual action

   Detection

   At 14:43 UTC alerts fired because the misbehaving switch was unreachable to pings. Engineers then saw that the primary authentication database was overwhelmed.

   Remediation

   • Shed load from the primary authentication database by slowing SSL certificate pushes to the edge, emails and other features
   • Moved API traffic that could use read replicas to the secondary data center by hand
   • Sent authentication calls fully back to the primary data center when the dashboard got worse
   • Rebuilt the first database replica, which put itself back into service at 21:20 UTC and returned performance to normal
   • Ramped the turned-down services back up so asynchronous processing could catch up

   Action items

   • Remove the cluster management limitation that requires rebuilding all replicas after a primary is promoted
   • Redesign the user sessions system so dashboard traffic can be steered between data centers
   • Change the auto-remediation configuration so a database replica is not promoted so quickly
   • Tune each system involved to be more resilient to failures of its dependencies

   Lessons

   Every system in the chain had redundancy and none failed completely; each only degraded, which made the chain of events much harder to model and foresee. Automating replica promotion so that it could trigger on a minute's notice had been a point of pride, but for at least one database a fast promotion can do more harm than the failure it answers. Most general-purpose cluster managers use Raft or Paxos rather than Byzantine fault tolerant protocols, accepting a rare failure mode in exchange for a protocol that is simpler to implement correctly and debug.

   View case

By root cause9