Incident Library
Applied filters ComponentDatabaseSymptomPacket lossFactorAutomation amplificationClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0347 About 25% of Scaleway FR-PAR-1 hypervisors lost VPC connectivity when both VPC BGP route reflectors got stuck
    Scaleway2024
    Symptoms

    On 18 October 2024, from 7:50 to 8:01 UTC, about 25% of the hypervisors in Scaleway's FR-PAR-1 availability zone lost VPC connectivity with the rest of the region, and products that use VPC failed with them. Public internet connectivity and Elastic Metal VPC connectivity were not affected.

    Network unreachablePacket lossNode NotReady
    Trigger

    At 6:20 UTC a Top of Rack switch in FR-PAR-1 had a software crash. One of the two BGP route reflectors for VPC ran on a hypervisor in that rack and was left in an abnormal state.

    Root cause
    Software defect
    Component
    Network control plane (SDN)BGP routingNetwork switchDatabaseManaged Kubernetes control plane and provider API
    Contributing factor
    Excessive blast radiusAutomation amplification
    Detection

    Health-check monitoring on both route reflectors detected the anomaly, and the Scaleway Customer Excellence and SRE teams detected the customer impact and opened an incident.

    Remediation
    • Fixed both VPC route reflectors by manual action
    Action items
    • Fixed the autohealing mechanism that had failed to fix route reflectors stuck in a stale state; the fix is https://github.com/FRRouting/frr/pull/17163
    • Run different software versions on the route reflectors so a single bug cannot stop several instances
    • Investigate the software issue that left the route reflectors in a stale state
    View case
  2. IL-0240 Cloudflare API requests failed and the dashboard slowed for hours after the authentication database lost its replicas
    Cloudflare2020
    Symptoms

    On 2 November 2020 the Cloudflare API and dashboard were degraded for six hours and 33 minutes. The success rate of API queries dropped at times to 75%, and the dashboard was up to 80 times slower than normal, while Cloudflare's edge network kept working.

    High error rateHigh latencyDatabase overloadWrites rejectedPacket loss
    Trigger

    At 14:43 UTC a network switch in a control plane data center partly failed: it kept LACP and BGP running but stopped forwarding some packets, and it recovered on its own six minutes later.

    Root cause
    DNS or network fault
    Component
    NIC, cable and switch port linkCoordination and key-value storeDatabase failover managerDatabaseDatabase replication
    Contributing factor
    Automation amplificationNo headroomHarmful manual action
    Detection

    At 14:43 UTC alerts fired because the misbehaving switch was unreachable to pings. Engineers then saw that the primary authentication database was overwhelmed.

    Remediation
    • Shed load from the primary authentication database by slowing SSL certificate pushes to the edge, emails and other features
    • Moved API traffic that could use read replicas to the secondary data center by hand
    • Sent authentication calls fully back to the primary data center when the dashboard got worse
    • Rebuilt the first database replica, which put itself back into service at 21:20 UTC and returned performance to normal
    • Ramped the turned-down services back up so asynchronous processing could catch up
    Action items
    • Remove the cluster management limitation that requires rebuilding all replicas after a primary is promoted
    • Redesign the user sessions system so dashboard traffic can be steered between data centers
    • Change the auto-remediation configuration so a database replica is not promoted so quickly
    • Tune each system involved to be more resilient to failures of its dependencies
    Lessons

    Every system in the chain had redundancy and none failed completely; each only degraded, which made the chain of events much harder to model and foresee. Automating replica promotion so that it could trigger on a minute's notice had been a point of pride, but for at least one database a fast promotion can do more harm than the failure it answers. Most general-purpose cluster managers use Raft or Paxos rather than Byzantine fault tolerant protocols, accepting a rare failure mode in exchange for a protocol that is simpler to implement correctly and debug.

    View case
Check your cluster for this

These 2 cases involve Database and BGP routing. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause2

Software defect1 case

Component
BGP routingDatabaseManaged Kubernetes control plane and provider APINetwork control plane (SDN)Network switch
Contributing factor
Automation amplificationExcessive blast radius
Organisation
Scaleway
  1. IL-0347 About 25% of Scaleway FR-PAR-1 hypervisors lost VPC connectivity when both VPC BGP route reflectors got stuck
    Scaleway2024

DNS or network fault1 case

Component
Coordination and key-value storeDatabaseDatabase replicationDatabase failover managerNIC, cable and switch port link
Contributing factor
Automation amplificationHarmful manual actionNo headroom
Organisation
Cloudflare
  1. IL-0240 Cloudflare API requests failed and the dashboard slowed for hours after the authentication database lost its replicas
    Cloudflare2020