Incident Library

Component

Network control plane (SDN)

네트워크 컨트롤 플레인 · network-control-plane

12 cases · 2017 to 2026

Root cause Software defect5Faulty deployment4Capacity exhaustion1Configuration change1Dependency failure1
Contributing factor Excessive blast radius8Gap in automated tests3Automation amplification2Monitoring blind spot2Ineffective rollback2Unverified assumption1Circular dependency1Change with a delayed effect1Ignored early signal1Limited debug access1No alert on limit1No safeguard on destructive change1No staged rollout1Overlapping failures1Partial rollout1Unsafe failure behaviour1
Symptom Network unreachable10Packet loss6High latency3Service outage3Cluster or node provisioning stall2High error rate2Scale-up failure1DNS resolution failures1Intermittent connection failures1Intermittent timeouts1Link flapping1Monitoring outage1Node NotReady1OOMKilled1Probe failures1Misrouted requests1Stale endpoint routing1
Organisation Google Cloud4Fly.io1IONOS Cloud1Linode1Microsoft Azure1Nebius1Packet1Railway1Scaleway1

Cases, by severity

  1. IL-0356 Every domain hosted on Railway returned HTTP 404 to new connections for five minutes because route lookups failedRailway · 2026

    Root cause Faulty deployment

  2. IL-0354 Fly.io private networking (6PN) broke for some machines after edges received gateway /80 routesFly.io · 2026

    Root cause Faulty deployment

  3. IL-0350 Hosts in an IONOS TXL cluster lost connectivity when an InfiniBand fabric resweep failed on a full multicast tableIONOS Cloud · 2026

    Root cause Capacity exhaustion

  4. IL-0563 Linode servers lost connectivity in all data centers as route servers built incomplete BGP route tablesLinode · 2025

    Root cause Faulty deployment

  5. IL-0233 A VPC control plane release broke IP alias routes in Nebius eu-west1, and Managed Kubernetes pods lost connectivityNebius · 2025

    Root cause Software defect

  6. IL-0347 About 25% of Scaleway FR-PAR-1 hypervisors lost VPC connectivity when both VPC BGP route reflectors got stuckScaleway · 2024

    Root cause Software defect

  7. IL-0339 Azure Front Door and Azure CDN users saw intermittent connection errors, timeouts and latency spikes for hoursMicrosoft Azure · 2024

    Root cause Configuration change

  8. IL-0529 New Google Cloud VMs and other instances came up without network connectivity in all zones after VPC controllers stoppedGoogle Cloud · 2024

    Root cause Software defect

  9. IL-0530 Virtual router controllers in us-central1 ran out of memory in a loop, cutting networking for new and migrated VMsGoogle Cloud · 2023

    Root cause Faulty deployment

  10. IL-0358 Packet Public Cloud locations had high latency and unreachable paths from routing loops and blackholed trafficPacket · 2019

    Root cause Dependency failure

  11. IL-0222 Google Cloud US regions had heavy packet loss and some went unreachable after the network control plane was descheduledGoogle Cloud · 2019

    Root cause Software defect

  12. IL-0396 New or live-migrated GCE instances, VPN tunnels and network load balancers were unreachable on public IPs for over 2hGoogle Cloud · 2017

    Root cause Software defect

View on the graph