Incident Library
Applied filters SymptomNetwork unreachableFactorUnverified assumptionClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases3 cases

1 to 3 of 3

  1. IL-0354 Fly.io private networking (6PN) broke for some machines after edges received gateway /80 routes
    Fly.io2026
    Symptoms

    On 23 September 2026, private networking (6PN) stopped working for a subset of Fly.io machines. Fly.io mitigated it in about 10 minutes, and the problem came back briefly the following day.

    Network unreachableMisrouted requests
    Trigger

    Fly.io changed its 6PN routing so that edges act as gateways for non-static peers. Every edge was then treated as a gateway and got a /80 route for its prefix in the Linux routing table, which overrode the per-machine routing for machines that had kept a reused host ID.

    Root cause
    Faulty deployment
    Component
    Host routes and routing rulesNetwork control plane (SDN)
    Contributing factor
    Unverified assumptionGap in automated tests
    Remediation
    • Deleted the bad /80 routes from the routing table on all affected hosts
    • Rolled back the changes to s6pnforwarderd, the daemon that manages 6PN routes
    • Shipped a fix that adds interactive peers only as /128 routes, which did not prevent a brief recurrence the next day
    • Changed routing to always look first for machines that carry a host's ID, whatever kind of host it is
    Action items
    • Give the machines used to test and alert on 6PN non-local 6PN addresses
    • Make host IDs in 6PN addresses carry no weight in routing decisions, for gateway peers as well as machines
    • Move new peers and machines across the board to a better model than the current hack kept for existing gateway peers
    Lessons

    Not everyone at Fly.io knew that a worker's host ID could be reused by a host that is not a worker, because host ID assignment was not well documented, and the gateways had not been touched for a long time, so the problem stayed hidden until edges became gateways. All existing 6PN checks used machines on their original host, so they could not catch any problem with host ID reuse or stable 6PN addresses; host IDs need to stop influencing 6PN routing in general for this kind of incident not to recur.

    View case
  2. IL-0473 Jenkins infrastructure went fully down after a pod IP range fix deleted every AKS node pool and could not recreate them
    Jenkins project2023
    Symptoms

    On 7 July 2023 from 11:05 to 15:25 UTC many public Jenkins infrastructure services were completely down, including get.jenkins.io, plugins.jenkins.io, accounts.jenkins.io, javadoc.jenkins.io and incrementals.jenkins.io, while ci.jenkins.io, issues.jenkins.io, repo.jenkins-ci.org and www.jenkins.io were partly or completely disrupted.

    Network unreachableResources deletedCluster or node provisioning stallService outage
    Trigger

    After the upgrade to Kubernetes 1.25 changed the range of the pods' originating IP and LDAP access was blocked, a fix (Azure PR 431) deployed at 10:55 UTC to set a proper set of pod IP ranges removed all node pools of the cluster.

    Root cause
    Configuration change
    Component
    Node poolCNI pluginCloud load balancer
    Contributing factor
    Unverified assumptionNo staged rolloutNo safeguard on destructive change
    Detection

    After the upgrade, at 10:30 UTC, the team noticed that services inside the cluster, such as accounts.jenkins.io, could not reach the LDAP service.

    Remediation
    • Created the cluster again from scratch with a fixed network setup
    • Set a custom pod CIDR in a range separate from the virtual network
    • Moved the services and their DNS records to new public IPs
    • Pushed a pull request that recorded the manual rebuild, including the IP setup, in code
    Action items
    • Protect the public IPs from deletion with a Management Lock
    • Keep the public IPs in a separate resource group and point the Kubernetes-managed load balancers to it with the service.beta.kubernetes.io/azure-load-balancer-resource-group annotation
    • Improve network diagrams and documentation so possible overlaps are visible when preparing operations
    • Avoid changing the configuration of all AKS node pools at once
    Lessons

    The team said that changing one node pool at a time would have shown the problem after the first pool and avoided a full outage, and that the public IPs should change as little as possible because users behind corporate firewalls with allow-lists depend on them. The team also noted that it could rebuild, in under 4 hours, the cluster that hosts almost all public services.

    View case
  3. IL-0225 An IGP purge command made every Microsoft WAN router recompute routes, causing global latency, timeouts and packet loss
    Microsoft Azure2023
    Symptoms

    From 07:08 UTC on 25 January 2023, customers saw long network latency and timeouts when connecting to resources in Azure regions and to other Microsoft services such as Microsoft 365 and Power Platform. Microsoft's monitoring detected DNS and WAN issues from 07:11 UTC.

    High latencyIntermittent timeoutsIntermittent connection failuresPacket lossNetwork unreachable
    Trigger

    A network engineer adding capacity to the global WAN in Madrid ran a step of a recently changed standard operating procedure that purged the IGP database; on the router manufacturer involved, that command runs on all IGP-joined routers instead of only the local one.

    Root cause
    Configuration change
    Component
    Backbone network (WAN)Interior routing (IGP)BGP routing
    Contributing factor
    No configuration testsUnverified assumptionExcessive blast radiusHarmful manual action
    Detection

    Monitoring detected DNS and WAN issues from 07:11 UTC and alerted the on-call engineers within 5 minutes of the command being run. The team reviewed all recent changes and identified the command by 08:20 UTC. The engineer making the change was not informed of the alerts because of the unqualified changes to the SOP.

    Remediation
    • Restarted by hand the paused WAN health and traffic engineering systems, which ended the remaining packet loss
    Action items
    • Audit and block similar widespread-impact commands across all three vendors for all WAN router roles
    • Give on-call engineers real-time visibility of approved and unqualified activity on network devices
    • Run regular mandatory operational training and attestation of following all SOPs
    • Have the Change Advisory Board review all SOPs still pending qualification within 30 days
    View case
Check your cluster for this

These 3 cases involve Backbone network (WAN) and BGP routing. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause2

Faulty deployment1 case

Component
Host routes and routing rulesNetwork control plane (SDN)
Contributing factor
Unverified assumptionGap in automated tests
Organisation
Fly.io
  1. IL-0354 Fly.io private networking (6PN) broke for some machines after edges received gateway /80 routes
    Fly.io2026

Configuration change2 cases

Component
Backbone network (WAN)1BGP routing1Cloud load balancer1CNI plugin1Interior routing (IGP)1Node pool1
Contributing factor
Unverified assumption2Excessive blast radius1Harmful manual action1No configuration tests1No safeguard on destructive change1No staged rollout1
Organisation
Jenkins project1Microsoft Azure1
  1. IL-0473 Jenkins infrastructure went fully down after a pod IP range fix deleted every AKS node pool and could not recreate them
    Jenkins project2023
  2. IL-0225 An IGP purge command made every Microsoft WAN router recompute routes, causing global latency, timeouts and packet loss
    Microsoft Azure2023