IL-0347 Scaleway · 2024-10-18networkingkubernetes
About 25% of Scaleway FR-PAR-1 hypervisors lost VPC connectivity when both VPC BGP route reflectors got stuck
Scaleway's VPC networks are a software-defined network on top of hypervisors, and BGP route reflectors are one of its building blocks. Critical systems run at least two active instances so that one can carry the whole load. At 6:20 UTC on 18 October 2024 a ToR switch crashed; its unstable software modules took about 50 seconds to move all rack traffic to the second ToR switch, and packets to and from the rack's hypervisors were dropped meanwhile. Traffic was fully rerouted by 6:21 UTC, and the crashed switch finished its automatic reboot at 6:38 UTC.
The instability spread to a VPC route reflector hosted on a hypervisor in that rack. Its software stack became stuck in an abnormal state that its auto-restart could not clear, and VPC kept working on the second route reflector without redundancy. At 7:50 UTC the second route reflector also became stuck in a non-operational state, and about 25% of the hypervisors lost VPC connectivity with the rest of the region. Health-check monitoring on both route reflectors detected the anomaly, but the auto-restart mechanism failed. Scaleway's Customer Excellence and SRE teams detected the impact and opened an incident, both route reflectors were fixed by hand, VPC connectivity was back by 8:01 UTC, and redundancy was back to normal by 8:07 UTC. Scaleway fixed the autohealing that had failed to repair route reflectors in a stale state, linking a pull request to FRRouting, and planned to run different software versions on the route reflectors so one bug cannot stop all of them.
- Severity
- –
- Scope
- VPC connectivity for about 25% of FR-PAR-1 hypervisors and the products built on it: Managed Database and Redis, Serverless Functions and Containers, Kubernetes Kapsule nodes and control planes; earlier, about a minute of heavy packet loss for instances in one rack
- Users affected
- Customers with resources on the affected hypervisors and in the affected rack
- Region
- FR-PAR-1
- Time to detect
- –
- Time to resolve
- 1h 41m
Symptoms
On 18 October 2024, from 7:50 to 8:01 UTC, about 25% of the hypervisors in Scaleway's FR-PAR-1 availability zone lost VPC connectivity with the rest of the region, and products that use VPC failed with them. Public internet connectivity and Elastic Metal VPC connectivity were not affected.
An earlier, shorter disturbance came first. At 6:20 UTC a Top of Rack (ToR) switch had a software crash, and for about 50 seconds, until traffic had failed over to the second ToR switch of the rack, instances in that single rack saw heavy packet loss on all kinds of network connectivity. Some managed database customers got 500 HTTP errors on connections to their databases during that minute. The status page later described the service as degraded since 08:20 AM CEST and down between 09:49 and 10:04 AM CEST.
During the VPC outage, customers of Managed Database and Redis could not reach their databases over VPC. One serverless compute node was unavailable, and some Serverless Functions and Containers customers could get 500 errors while their workloads were rescheduled. In Kubernetes Kapsule, a network partition between nodes stopped applications on different nodes from communicating, some control planes were unavailable because the infrastructure that hosts them uses VPC, and autohealing replaced some nodes that could not report their availability, so workloads were rescheduled or restarted. By 8:04 UTC almost all Kapsule clusters had recovered.
Root cause
Root cause Software defect
- Trigger
- At 6:20 UTC a Top of Rack switch in FR-PAR-1 had a software crash. One of the two BGP route reflectors for VPC ran on a hypervisor in that rack and was left in an abnormal state.
Detection
Health-check monitoring on both route reflectors detected the anomaly, and the Scaleway Customer Excellence and SRE teams detected the customer impact and opened an incident.
Remediation
- Fixed both VPC route reflectors by manual action
Action items
- Fixed the autohealing mechanism that had failed to fix route reflectors stuck in a stale state; the fix is https://github.com/FRRouting/frr/pull/17163
- Run different software versions on the route reflectors so a single bug cannot stop several instances
- Investigate the software issue that left the route reflectors in a stale state
Lessons
–
Sources
- https://status.scaleway.com/incidents/g3q3c4d43gkb · official-status · archived copy
References
Related cases
IL-0233A VPC control plane release broke IP alias routes in Nebius eu-west1, and Managed Kubernetes pods lost connectivityNebius · 2025same root cause · shared Network control plane (SDN), BGP routing
IL-0222Google Cloud US regions had heavy packet loss and some went unreachable after the network control plane was descheduledGoogle Cloud · 2019same root cause · shared Network control plane (SDN), BGP routing
IL-0451Setting indexes INVISIBLE on Aurora MySQL 3.10 spiked CPU and ReadIOPS on all three readers but not on the writerDaangn · 2026same root cause · shared Database
IL-0313Decagon voice requests stalled for exactly 300 seconds behind self-managed PgBouncer and exhausted the SQLAlchemy poolDecagon · 2026same root cause · shared Database
IL-0281PostHog's Logs ClickHouse cluster silently deleted almost all its S3 data, losing US logs older than 3 daysPostHog · 2026same root cause · shared Database
IL-0310An Aurora PostgreSQL failover to an upsized reader reversed twice and backend writes failed in read-only transactionsHightouch · 2025same root cause · shared Database
IL-0150A latent race condition in DynamoDB's DNS automation left an empty DNS record for dynamodb.us-east-1.amazonaws.com, and the DynamoDB and EC2 launch failures that followed caused EKS, ECS and Fargate container launch failures and cluster scaling delays in us-east-1Amazon Web Services · 2025same root cause · shared Database
IL-0436A supremum lock left by ROLLBACK TO SAVEPOINT in a long transaction blocked inserts and filled Aurora MySQL connectionsfreee · 2025same root cause · shared Database
IL-0527OpenStreetMap.org made no replication diffs for 16 hours as PostgreSQL logical decoding failed on a memory alloc errorOpenStreetMap Foundation · 2025same root cause · shared Database
IL-0329A routine migration left PGAudit stuck holding Postgres locks, causing intermittent timeouts and a 2-minute outageincident.io · 2025same root cause · shared Database
IL-0373Heroku Postgres Essential-tier databases in the EU region ran out of memory and refused connections, twice in four daysHeroku · 2024same root cause · shared Database
IL-0529New Google Cloud VMs and other instances came up without network connectivity in all zones after VPC controllers stoppedGoogle Cloud · 2024same root cause · shared Network control plane (SDN)
IL-0024Expensive cache-backed API calls and uncancelled database requests overloaded MongoDB and took the Hugging Face Hub down three timesHugging Face · 2024same root cause · shared Database
IL-0342DigitalOcean's AMS3, LON1 and SFO2 data centers lost Internet connectivity in repeated waves as edge routers failedDigitalOcean · 2023same root cause · shared BGP routing
IL-0217Honeycomb's main MySQL database locked up under cache-miss reads and ran out of connections, a 68-minute total outageHoneycomb · 2023same root cause · shared Database
IL-0330Requests timed out for two weeks waiting on Go's database/sql connection pool, one for almost 20 secondsincident.io · 2023same root cause · shared Database
IL-0385GitHub MySQL read replicas hit a semaphore deadlock during a schema migration's table rename and kept crash-recoveringGitHub · 2021same root cause · shared Database
IL-0462Some MySQL pods on Shopify's GKE clusters took up to 30 minutes to start, slowing instance replacementShopify · 2021same root cause · shared Database
IL-0359Queries to Gandi LiveDNS timed out when two crashed anycast DNS nodes kept announcing all three name server addressesGandi · 2020same root cause · shared BGP routing
IL-0196Adding a second Redis replica and lowering memory limits in one RedisFailover change restarted the only Redis pod that held data, and Sentinel promoted the empty new replica, erasing the dataPalark · 2020same root cause · shared Database
IL-0486Figma's PostgreSQL 9 database on RDS slowed under high write IOPS and lock contention until the site was unavailableFigma · 2020same root cause · shared Database
IL-0241A database shard could not elect a new primary, and the Stripe API failed most requests in two periods on one dayStripe · 2019same root cause · shared Database
IL-0439Database access slowed after each block device and filesystem expansion because the Linux kernel dropped the page cacheCybozu · 2018same root cause · shared Database
IL-0388A Google Cloud live migration took a Discord Redis primary offline, and the failover broke API instances and then guildsDiscord · 2017same root cause · shared Database
IL-0396New or live-migrated GCE instances, VPN tunnels and network load balancers were unreachable on public IPs for over 2hGoogle Cloud · 2017same root cause · shared Network control plane (SDN)
IL-0454Baemin billing servers could not open new MariaDB connections at the start of a card-discount eventWoowa Brothers · 2016same root cause · shared Database
IL-0536Amazon SimpleDB in US East returned HTTP 500 to nearly all API calls for two hours after its nodes left the clusterAmazon Web Services · 2014same root cause · shared Database
IL-0485Dropbox went down when an OS upgrade script reinstalled active MySQL master-replica pairs and backup restores ran slowDropbox · 2014same root cause · shared Database
IL-0589Every Kickstarter MySQL replica on Amazon RDS stopped replicating on a duplicate-key error in the backings tableKickstarter · 2013same root cause · shared Database
IL-0263GitHub was slow and erroring all day and down for 18 minutes as links between access and aggregation switches saturatedGitHub · 2012same root cause · shared Network switch
IL-0303GitHub's MySQL master failed over to a node with a cold buffer pool during a schema migration and GitHub.com went downGitHub · 2012same root cause · shared Database
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0347
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0347/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.