IL-0348 Scaleway · 2025-03-10networking

Scaleway's FR-PAR region lost public internet connectivity when one 100 Gbps backbone link was saturated

On 10 March 2025 Scaleway's network team was connecting a new data center to its backbone in the FR-PAR region. A mistake in a router configuration made during that deployment sent more than 1 Tbps of traffic between OpCore DC3 and DC5 over a single 100 Gbps link at 16:23 UTC. The link saturated, most internet traffic was effectively blackholed, and public internet connectivity between FR-PAR-2 and the rest of the region was severely disrupted. The error was detected quickly and rolled back within four minutes; the correct configuration was in place and the network had converged by 16:27 UTC.

The network outage lasted five minutes, but rollback and switchover mechanisms in other products had activated against a zone that was entirely cut off, and their back and forth kept the network congested and many products unstable for longer. Object Storage, a regional product, stopped replicating between zones during the outage; a bug in its software let replication run without a rate limit afterwards, which saturated links and kept the service degraded until about 16:50 UTC. DNS recovered at 17:20 UTC, Instances at 17:10 UTC, and Object Storage and Kubernetes Kapsule at 18:00 UTC according to the product list.

Severity: –
Scope: Public internet connectivity for all products in FR-PAR, with DNS, Object Storage, Kubernetes Kapsule, Instances, Transactional Email, Dedibox servers in DC5 and Elastic Metal in FR-PAR-2 affected; VPC and Dedibox RPN were not affected
Users affected: –
Region: FR-PAR (fr-par-1, fr-par-2, fr-par-3 and DC5)
Time to detect: –
Time to resolve: 1h 38m

Symptoms

On 10 March 2025 at 16:22 UTC, public internet connectivity in Scaleway's FR-PAR region broke for all products. Most internet traffic between the FR-PAR-2 availability zone and the rest of the PAR region was blackholed for about five minutes, while VPC traffic and Dedibox RPN kept working.

Products stayed broken for different lengths of time: authoritative and cache DNS from 16:23 to 17:20 UTC, Object Storage from 16:23 to 18:00 UTC, Kubernetes Kapsule from 16:23 to 18:00 UTC, virtual Instances from 16:23 to 17:10 UTC, and Transactional Email, Dedibox dedicated servers in DC5 and Elastic Metal in FR-PAR-2 from 16:23 to 16:27 UTC. A status page update posted at 18:19 CET said public connectivity in PAR2/DC5 was almost fully blackholed between 17:22 and 17:28.

Recovery took from minutes to tens of minutes after connectivity returned at 16:27 UTC. Rollback and switchover mechanisms in other products activated, but because the whole FR-PAR-2 zone was cut off they could not succeed, and their going back and forth heavily congested the network and made many products unstable. In Object Storage, replication between availability zones stopped during the outage and then ran without a rate limit once connectivity came back, saturating some links and causing latency and packet drops; the service was severely degraded across FR-PAR for about 25 minutes, until about 16:50 UTC. No customer data was lost.

Symptom Packet loss Service outage High latency

Root cause

Root cause Operator error

Component Backbone network (WAN)Object storage Authoritative DNS servers and records

Contributing factor Automation amplification

Trigger: While Scaleway's network team connected a new data center to the FR-PAR backbone, a human mistake in a router configuration rerouted more than 1 Tbps of traffic between OpCore DC3 and DC5 through a single 100 Gbps link.

Detection

The router configuration error was detected quickly and rolled back within four minutes.

Remediation

Rolled back the router configuration, restoring the correct configuration at 16:27 UTC

Action items

Harden the peer review and production communication process for changes
Add failsafes and tests before network configuration is applied at backbone level
Deploy the fix for the Object Storage synchronisation software issue

Lessons

–

Sources

https://status.scaleway.com/incidents/162zw5zd9x8r · official-status · archived copy

Related cases

IL-0221 A backbone maintenance command disconnected all Facebook data centers, and its DNS servers withdrew their BGP routesMeta · 2021
same root cause · shared Backbone network (WAN), Authoritative DNS servers and records
IL-0378 All Cloudflare R2 object writes and about 35% of reads failed when the R2 Gateway could not authenticate to storageCloudflare · 2025
same root cause · shared Object storage
IL-0531 Azure Database for MySQL Single Server and MariaDB in West US rejected every new connection for a batch of serversMicrosoft Azure · 2024
same root cause · shared Authoritative DNS servers and records
IL-0534 Amazon S3 in US-EAST-1 could not serve GET, LIST, PUT or DELETE requests for hours while its index subsystem restartedAmazon Web Services · 2017
same root cause · shared Object storage

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Backbone network (WAN), Object storage, Authoritative DNS servers and records among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0348
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0348/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.