Incident Library
Applied filters Root causeConfiguration changeSymptomPacket lossFactorMonitoring blind spotClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases3 cases

1 to 3 of 3

  1. IL-0339 Azure Front Door and Azure CDN users saw intermittent connection errors, timeouts and latency spikes for hours
    Microsoft Azure2024
    Symptoms

    From 11:45 UTC on 30 July 2024, a subset of customers got intermittent connection errors, timeouts and latency spikes when connecting to Microsoft services that use Azure Front Door (AFD) and Azure Content Delivery Network (CDN). The Azure Portal team saw the first service degradation at 11:47 UTC.

    Intermittent connection failuresIntermittent timeoutsHigh latencyPacket loss
    Trigger

    Around 11:45 UTC the Azure network DDoS protection service was disengaging after a routine DDoS mitigation and returning traffic to its default routing to AFD, but the routes at one site in Europe could not be updated because the network control plane could not reach that site during a local power outage.

    Root cause
    Configuration change
    Component
    CDN and edge networkDDoS protection serviceNetwork control plane (SDN)Data center power and cooling
    Contributing factor
    Overlapping failuresExcessive blast radiusMonitoring blind spot
    Detection

    The Azure Portal team detected service degradation at 11:47 UTC and began to investigate. At 12:10 UTC network monitoring correlated the Portal incident to a network issue at one site in Europe, and networking engineers joined.

    Remediation
    • Rerouted traffic under the standard congestion playbook, which removed most of the impact by 13:58 UTC
    • Ordered the network control plane to reissue its commands to the misbehaving device, which could not be reached
    • Removed the device that kept attracting traffic from the network and scanned the network for other issues
    Action items
    • Add the missing configuration on network devices so a DDoS mitigation issue in one geography cannot spread to another
    • Improve validation and monitoring in the Azure network to detect invalid configurations
    • Improve monitoring for when the DDoS protection service is unreachable from the control plane but still serving traffic
    View case
  2. IL-0461 Every rollout of Datadog's metrics query service on Kubernetes raised errors that the logs blamed on DNS
    Datadog2022
    Symptoms

    From September 2021, each rollout of an update to Datadog's metrics query service, which serves real-time and historical metric data to the web frontend and to monitor evaluation, came with a rise in errors. Clients retried failed queries, so few errors reached users, but the retries slowed dashboards and monitors. The service's logs showed DNS errors stopping it from connecting to the data stores it depends on.

    High error rateDNS resolution failuresHigh latencyOOMKilledPacket lossStale endpoint routing
    Trigger

    Rollouts of the metrics query service, which delete the old pods. The client behaviour behind the failures came from an August 2021 rollout of the metrics query clients that applied round_robin as the new default gRPC load balancing policy.

    Root cause
    Configuration change
    Component
    conntrack tableCiliumLinux kernelCoreDNSClient-side DNS resolver and cacheexternal-dnsHTTP and RPC client library
    Contributing factor
    Change with a delayed effectRetry stormUnknown limitMonitoring blind spot
    Detection

    Errors rose during rollouts of the metrics query service, and the service's logs showed DNS errors when connecting to its data stores.

    Remediation
    • Raised the memory of the node-local-dns pods from 64 MB to 256 MB, which stopped the OOM kills but not the DNS errors
    • Tested larger instance types and confirmed that scaling up the service's infrastructure resolved the errors
    • Removed the custom gRPC reconnect parameters from the clients to restore the defaults
    Action items
    • Contribute a change to Cilium that adds an unreachable route for a deleted pod's IP so clients get an ICMP error
    Lessons

    The errors looked like DNS but were not. ENA metrics and VPC Flow Logs gave the visibility into VPC connection tracking that host metrics did not, and Cilium, Kubernetes and AWS networking interact in ways that can make reverse path filtering drop packets it should allow. DNS propagation time depends on many settings. The causes were changes Datadog had made itself, to work around a dependency bug, to resolve earlier incidents and to override default reconnect parameters, which looked safe at the time and showed their effects much later, when the link was hard to see.

    View case
  3. IL-0361 Job searches on seek.com.au failed for 16 minutes as a Transit Gateway silently dropped 9001-byte packets
    SEEK2021
    Symptoms

    In May 2021 some users of seek.com.au got errors when they searched for jobs, for 16 minutes. SEEK was in the middle of moving the traffic between two VPCs, the one that carried searches, from a VPC peering connection to an AWS Transit Gateway, and the team noticed quickly that search was broken.

    High error ratePacket lossIntermittent timeouts
    Trigger

    In the second step of the migration, SEEK changed the routing table of the first VPC to use the transit gateway while the second VPC still routed over the peering connection, so the two directions of each connection took different paths.

    Root cause
    Configuration change
    Component
    Transit gateway or network interconnect
    Contributing factor
    Unknown limitUnreviewed defaultsMonitoring blind spot
    Detection

    The team noticed soon after the routing change that search on seek.com.au was not working.

    Remediation
    • Rolled back the migration of the search traffic's peering connection to the transit gateway
    Action items
    • Lower the MTU of EC2 instances to 8500, on Amazon Linux 2 with a script at /sbin/ifup-local that runs whenever a network interface comes up
    Lessons

    A transit gateway has a smaller MTU than a same-region peering connection, drops oversized packets silently and clamps the MSS only on packets that cross it. When a migration leaves the two directions of a TCP connection on different paths, the clamp lands on the wrong side of the handshake and large packets are lost; the fix is not to send packets larger than 8500 bytes.

    View case
Check your cluster for this

These 3 cases involve CDN and edge network and Cilium. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Configuration change3 cases

Component
CDN and edge network1Cilium1conntrack table1CoreDNS1Data center power and cooling1DDoS protection service1Client-side DNS resolver and cache1external-dns1HTTP and RPC client library1Transit gateway or network interconnect1Linux kernel1Network control plane (SDN)1
Contributing factor
Monitoring blind spot3Unknown limit2Excessive blast radius1Unreviewed defaults1Change with a delayed effect1Overlapping failures1Retry storm1
Organisation
Datadog1Microsoft Azure1SEEK1
  1. IL-0339 Azure Front Door and Azure CDN users saw intermittent connection errors, timeouts and latency spikes for hours
    Microsoft Azure2024
  2. IL-0461 Every rollout of Datadog's metrics query service on Kubernetes raised errors that the logs blamed on DNS
    Datadog2022
  3. IL-0361 Job searches on seek.com.au failed for 16 minutes as a Transit Gateway silently dropped 9001-byte packets
    SEEK2021