Incident Library

Applied filters SymptomNetwork unreachableClear allEvery case under the filters, with no sentence; newest first.

Suggested filters Search, and the values found in your words appear here.

Graph

Root cause
Component
Symptom
Contributing factor
Case

Cases39 cases

9 to 13 of 39

IL-0233 A VPC control plane release broke IP alias routes in Nebius eu-west1, and Managed Kubernetes pods lost connectivity
Nebius2025
Symptoms
On 13 March 2025 practically every pod of Nebius Managed Services for Kubernetes in the eu-west1 region lost network connectivity for at least some time between 13:36 and 18:11 UTC. Packets for the affected IP alias routes were not processed by the data plane. The eu-north1 region was not affected.
Network unreachable
Trigger
A release of a VPC control plane component in eu-west1, deployed to canaries from 13:36 UTC and to all other nodes from 14:35 UTC, changed route distinguisher values for a new feature. The change made a long-standing bug in the handling of route announcements take effect for the update messages of Managed Kubernetes pods.
Root cause
Software defect
Component
Network control plane (SDN)BGP routing
Contributing factor
Ineffective rollback Gap in automated tests Monitoring blind spot
Remediation
Restarted the host VPC component on the affected nodes one node at a time
Restarted the host VPC component on the remaining nodes in parallel once the restart was known to fix the problem
Action items
Fix the bug in the VPC host component itself
Add an alert on a diverging number of routes
Add an alert on the number of drops in the data plane
Develop a way to reset endpoints without restarting the host VPC component
Move the VPC's persistent e2e test virtual machines to canary hosts
Add persistent e2e tests for Managed Services for Kubernetes
Run e2e tests more often
Add a deployment stage in which non-critical internal services receive updates first
Lessons
The failure needed both an existing IP alias at release time and a particular order of node updates, and neither the testing installation nor the canaries met both, so neither caught it. Missing alerts on technical metrics that deviate from normal, here the number of routes and the number of drops, made the response much slower.
View case
IL-0526 OpenStreetMap.org went offline, then read-only, for 56 hours after HE.net routing equipment cut both fibre uplinks
OpenStreetMap Foundation2024
Symptoms
At 04:01 UTC on 15 December 2024, StatusCake monitoring sent SMS alerts that OpenStreetMap.org was unreachable. The website and the mapping API had gone completely offline at 03:53 UTC.
Network unreachable Service outage
Trigger
HE.net's routing equipment in Amsterdam failed at 03:53 UTC on 15 December 2024.
Root cause
Dependency failure
Component
ISP and transit networks Database replication
Contributing factor
Single point of failure No runbook
Detection
StatusCake monitoring detected the outage at 04:01 UTC and sent SMS alerts, and the incident was reported to the operations team at once.
Remediation
Activated a read-only backup instance of OpenStreetMap.org in Dublin
Brought a new Equinix Internet link into service in Amsterdam, bypassing HE.net and restoring full functionality
HE.net replaced its routing hardware and restored its service, and routing through HE.net resumed
Action items
Move to a multi-ISP architecture so another provider can take over when one fails
Research backup ISPs for the Amsterdam servers, including BGP and keeping HE.net as a fallback
Add deeper route and latency monitoring and an automated escalation protocol
Hold failover drills for switching between ISPs and keep replication up to date
Agree SLAs with HE.net and Equinix Internet with defined escalation procedures and response times
Prepare the 4G out-of-band system as a manual fallback uplink that can sync Postgres replication data, and upgrade its hardware
Document outage communication procedures and delegate communication during outages
Investigate decoupling the OAuth service so authentication keeps working while the database is read-only
Lessons
Relying on HE.net alone left the Amsterdam servers with a single point of failure, so OpenStreetMap will keep multiple upstream providers. The 4G out-of-band link proved invaluable and should be able to carry data synchronisation. Resumed replication would have restored editing faster than the read-only backup. Frequent contact with the ISP gave no useful estimates, so the SLA needs to be better understood. Alerting on upstream connectivity needs refining, and a read-only OpenStreetMap.org also blocked logins to services that were not affected, because it could not save new tokens.
View case
IL-0347 About 25% of Scaleway FR-PAR-1 hypervisors lost VPC connectivity when both VPC BGP route reflectors got stuck
Scaleway2024
Symptoms
On 18 October 2024, from 7:50 to 8:01 UTC, about 25% of the hypervisors in Scaleway's FR-PAR-1 availability zone lost VPC connectivity with the rest of the region, and products that use VPC failed with them. Public internet connectivity and Elastic Metal VPC connectivity were not affected.
Network unreachable Packet loss Node NotReady
Trigger
At 6:20 UTC a Top of Rack switch in FR-PAR-1 had a software crash. One of the two BGP route reflectors for VPC ran on a hypervisor in that rack and was left in an abnormal state.
Root cause
Software defect
Component
Network control plane (SDN)BGP routing Network switch Database Managed Kubernetes control plane and provider API
Contributing factor
Excessive blast radius Automation amplification
Detection
Health-check monitoring on both route reflectors detected the anomaly, and the Scaleway Customer Excellence and SRE teams detected the customer impact and opened an incident.
Remediation
Fixed both VPC route reflectors by manual action
Action items
Fixed the autohealing mechanism that had failed to fix route reflectors stuck in a stale state; the fix is https://github.com/FRRouting/frr/pull/17163
Run different software versions on the route reflectors so a single bug cannot stop several instances
Investigate the software issue that left the route reflectors in a stale state
View case
IL-0244 A queued pull request destroyed the EKS subnet route table associations and Cloud Platform services went offline
UK Ministry of Justice2024
Symptoms
On 20 September 2024 at 11:24 the Cloud Platform team received high priority Pingdom alerts for services on the live cluster, and from 11:26 teams reported in the #ask channel that services were unavailable. Users reported that services could not be resolved.
Service outage Network unreachable Resources deleted
Trigger
At 11:21 the infrastructure-vpc-live-1 pipeline was unpaused, and a pull request that had been waiting in its queue ran automatically.
Root cause
Operator error
Component
Terraform CI system VPC route tables and firewall rules
Contributing factor
No safeguard on destructive change
Detection
High priority Pingdom alerts for live cluster services reached the Cloud Platform team at 11:24, and users reported that services could not be resolved.
Remediation
Ran terraform apply locally to rebuild the subnet route table associations
Action items
Review and update the process for pausing and resuming infrastructure pipelines so every team member knows its implications
Find a way to hold queued pull requests while manual infrastructure updates are under way
Find ways to isolate the plan and apply tasks of infrastructure pipelines better
View case
IL-0423 A core switch failure in Paperspace's NY2 region cut VMs off their network file system and left them read-only
Paperspace2024
Symptoms
On 16 June 2024, from around 10:30 UTC, users could not interact with Gradient notebooks and Core VMs in Paperspace's NY2 region. A large number of virtual machines lost network access to the Network File System they resided on and went into Read-Only (RO) mode, so customers could not write to them. Network connectivity came back by 18:20 UTC, but the affected VMs stayed unavailable in RO mode until the next day, and after Core was fixed the Gradient platform still ran with degraded performance while it recovered.
Storage disconnected or stale mounts Network unreachable Service outage
Trigger
A core switch in NY2 failed, and the resulting spike in traffic degraded network performance.
Root cause
DNS or network fault
Component
Shared network filesystem
Contributing factor
–
Remediation
Restored network connectivity to the affected VMs by 18:20 UTC on 16 June
Returned the affected VMs from Read-Only to Read-Write mode between 5:58 and 12:23 UTC on 17 June
Action items
Redesign the network
Install new equipment
View case

Check your cluster for this

These 39 cases involve BGP routing and Network control plane (SDN). kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap