Incident Library

Symptom

Packet loss

패킷 손실 · packet-loss

24 cases · 2012 to 2026

Root cause Capacity exhaustion5Configuration change5Software defect4Faulty deployment3Dependency failure2DNS or network fault2Cloud provider outage1Hardware failure1Operator error1
Component BGP routing7Backbone network (WAN)6Network control plane (SDN)6Cloud load balancer4Database4Transit gateway or network interconnect4CDN and edge network3Database replication2Client-side DNS resolver and cache2Linux kernel2NIC, cable and switch port link2API gateway1Cloud block storage1In-memory cache1Automated capacity manager1Cilium1CNI plugin1conntrack table1Coordination and key-value store1CoreDNS1Data center power and cooling1DDoS protection service1Authoritative DNS servers and records1Amazon ECS1external-dns1Database failover manager1HTTP and RPC client library1Interior routing (IGP)1InfiniBand fabric1Managed Kubernetes control plane and provider API1NAT gateway1Network switch1Object storage1
Contributing factor Excessive blast radius7Automation amplification6Monitoring blind spot5Unreviewed defaults3Change with a delayed effect3Harmful manual action3Overlapping failures3Circular dependency2Ignored early signal2Unknown limit2Limited debug access2No configuration tests2No headroom2Retry storm2Unrepresentative staging2Unverified assumption1Gray failure1Unpatched known bug1No alert on limit1No safeguard on destructive change1Ineffective rollback1
Organisation Google Cloud4Cloudflare3Microsoft Azure3Scaleway2Amazon Web Services1Canva1Cybozu1Datadog1incident.io1Korea Ministry of the Interior and Safety1Linode1Mailgun1Packet1Sakura Internet1SEEK1Slack1

Cases

  1. IL-0394 A Cloudflare router in Miami leaked IPv6 BGP routes to peers and providers for 25 minutes, congesting the backboneCloudflare · 2026

    Root cause Configuration change

  2. IL-0564 Linode's Miami, Milan and Sao Paulo data centers had packet loss as traffic moved onto congested alternate pathsLinode · 2025

    Root cause Capacity exhaustion

  3. IL-0341 Traffic between Cloudflare and AWS us-east-1 had high latency, packet loss and origin failures for almost four hoursCloudflare · 2025

    Root cause Capacity exhaustion

  4. IL-0348 Scaleway's FR-PAR region lost public internet connectivity when one 100 Gbps backbone link was saturatedScaleway · 2025

    Root cause Operator error

  5. IL-0257 canva.com went down for about 50 minutes when 1.5 million requests per second OOM-killed every API Gateway taskCanva · 2024

    Root cause Dependency failure

  6. IL-0347 About 25% of Scaleway FR-PAR-1 hypervisors lost VPC connectivity when both VPC BGP route reflectors got stuckScaleway · 2024

    Root cause Software defect

  7. IL-0440 Pods on Cybozu's Neco Kubernetes platform got frequent no such host and i/o timeout DNS errors from dropped packetsCybozu · 2024

    Root cause Capacity exhaustion

  8. IL-0339 Azure Front Door and Azure CDN users saw intermittent connection errors, timeouts and latency spikes for hoursMicrosoft Azure · 2024

    Root cause Configuration change

  9. IL-0529 New Google Cloud VMs and other instances came up without network connectivity in all zones after VPC controllers stoppedGoogle Cloud · 2024

    Root cause Software defect

  10. IL-0293 Cloud Bigtable in us-central1 had higher latency and connection errors for about 32 hours from low-rate packet lossGoogle Cloud · 2023

    Root cause Faulty deployment

  11. IL-0540 Korea's local government administrative services and Government24 went down as a router port dropped most large packetsKorea Ministry of the Interior and Safety · 2023

    Root cause Hardware failure

  12. IL-0530 Virtual router controllers in us-central1 ran out of memory in a loop, cutting networking for new and migrated VMsGoogle Cloud · 2023

    Root cause Faulty deployment

  13. IL-0386 After a move to Google Cloud, Postgres connection timeouts and memcached i/o timeouts spiked on GKE nodesincident.io · 2023

    Root cause Capacity exhaustion

  14. IL-0225 An IGP purge command made every Microsoft WAN router recompute routes, causing global latency, timeouts and packet lossMicrosoft Azure · 2023

    Root cause Configuration change

  15. IL-0532 Azure Software Load Balancer sent packets of one flow to different backends, dropping Firewall and Private Link trafficMicrosoft Azure · 2022

    Root cause Faulty deployment

  16. IL-0461 Every rollout of Datadog's metrics query service on Kubernetes raised errors that the logs blamed on DNSDatadog · 2022

    Root cause Configuration change

  17. IL-0361 Job searches on seek.com.au failed for 16 minutes as a Transit Gateway silently dropped 9001-byte packetsSEEK · 2021

    Root cause Configuration change

  18. IL-0336 AWS Direct Connect traffic into the Tokyo Region had packet loss and intermittent connectivity for about six hoursAmazon Web Services · 2021

    Root cause Software defect

  19. IL-0220 An overloaded AWS Transit Gateway dropped packets on 4 January 2021, and Slack's web tier saturated and went downSlack · 2021

    Root cause Cloud provider outage

  20. IL-0240 Cloudflare API requests failed and the dashboard slowed for hours after the authentication database lost its replicasCloudflare · 2020

    Root cause DNS or network fault

  21. IL-0222 Google Cloud US regions had heavy packet loss and some went unreachable after the network control plane was descheduledGoogle Cloud · 2019

    Root cause Software defect

  22. IL-0358 Packet Public Cloud locations had high latency and unreachable paths from routing loops and blackholed trafficPacket · 2019

    Root cause Dependency failure

  23. IL-0590 F5 load balancers cached a path MTU of 296 after Rackspace reboots, and Mailgun connections failed intermittentlyMailgun · 2014

    Root cause DNS or network fault

  24. IL-0537 Sakura Cloud host servers repeatedly lost their storage connection, about five minutes per port switchoverSakura Internet · 2012

    Root cause Capacity exhaustion

View on the graph