Cases39 cases

7 to 11 of 39

1. IL-0227 Most Heroku dynos in Private Spaces lost outbound connectivity after an automated OS update restarted host networking

   Heroku2025

   Symptoms

   From 06:00 UTC on 10 June 2025, Heroku services degraded badly: customers reported intermittent logins, applications failed, and Heroku's monitoring detected widespread errors with dyno networking. Heroku's own tools and the Heroku Status Page were affected as well, which delayed communication with customers and left many of them without accurate status updates. Only at 13:58 UTC did the team find a workaround to post updates to the @herokustatus account on X.

   Network unreachableService outageHigh error rate

   Trigger

   An automated operating system update ran on Heroku's production infrastructure when it should have been disabled, and it restarted the hosts' networking services.

   Root cause

   Breaking change in an upgrade

   Component

   Host routes and routing rules

   Contributing factor

   Unpinned versionCircular dependencyLimited debug access

   Detection

   Customers reported intermittent logins, and Heroku's monitoring detected widespread errors with dyno networking. Heroku's own tools and the Heroku Status Page were affected as well.

   Remediation

   • Restarted internal instances in rotation, which gave some relief
   • Had an upstream vendor invalidate the token used for the automated updates
   • Disabled the automated upgrade service
   • Recycled dynos across the whole fleet

   Action items

   • Put permanent controls in place so no system change happens outside the controlled deployment process
   • Audit all base images for similar risks
   • Make network routing survive graceful restarts of the networking service
   • Build fully independent backup communication channels
   • Overhaul incident response tooling, including tools that run queries across the whole fleet
   • Streamline break-glass procedures for rapid access to critical systems
   • Improve monitoring to detect complex issues earlier

   Lessons

   Heroku's post-mortem names three areas: production lacked immutability controls, so an automated process could change it; communication was too slow, made worse by a status page that went down with the platform; and gaps in tooling and process made recovery take longer than it should have.

   View case
2. IL-0563 Linode servers lost connectivity in all data centers as route servers built incomplete BGP route tables

   Linode2025

   Symptoms

   At about 16:38 UTC on 17 March 2025, internal alerts fired across multiple Linode data centers because various servers had been unreachable for an extended period. At the same time customers began reporting problems with several products.

   Network unreachableService outage

   Trigger

   A release of Akamai Compute's internal backend API component was rolling out, and during the rollout the API gave inconsistent responses to the route servers.

   Root cause

   Faulty deployment

   Component

   BGP routingNetwork control plane (SDN)

   Contributing factor

   Partial rollout

   Detection

   Internal alerts fired across multiple data centers showing servers unreachable for an extended period, and customers reported problems with multiple products at the same time.

   Remediation

   • Restarted the route servers in all data centers in phases, starting with the most affected regions

   Action items

   • Hold backend API releases until there is a better way to make sure route data for route servers is contiguous

   View case
3. IL-0233 A VPC control plane release broke IP alias routes in Nebius eu-west1, and Managed Kubernetes pods lost connectivity

   Nebius2025

   Symptoms

   On 13 March 2025 practically every pod of Nebius Managed Services for Kubernetes in the eu-west1 region lost network connectivity for at least some time between 13:36 and 18:11 UTC. Packets for the affected IP alias routes were not processed by the data plane. The eu-north1 region was not affected.

   Network unreachable

   Trigger

   A release of a VPC control plane component in eu-west1, deployed to canaries from 13:36 UTC and to all other nodes from 14:35 UTC, changed route distinguisher values for a new feature. The change made a long-standing bug in the handling of route announcements take effect for the update messages of Managed Kubernetes pods.

   Root cause

   Software defect

   Component

   Network control plane (SDN)BGP routing

   Contributing factor

   Ineffective rollbackGap in automated testsMonitoring blind spot

   Remediation

   • Restarted the host VPC component on the affected nodes one node at a time
   • Restarted the host VPC component on the remaining nodes in parallel once the restart was known to fix the problem

   Action items

   • Fix the bug in the VPC host component itself
   • Add an alert on a diverging number of routes
   • Add an alert on the number of drops in the data plane
   • Develop a way to reset endpoints without restarting the host VPC component
   • Move the VPC's persistent e2e test virtual machines to canary hosts
   • Add persistent e2e tests for Managed Services for Kubernetes
   • Run e2e tests more often
   • Add a deployment stage in which non-critical internal services receive updates first

   Lessons

   The failure needed both an existing IP alias at release time and a particular order of node updates, and neither the testing installation nor the canaries met both, so neither caught it. Missing alerts on technical metrics that deviate from normal, here the number of routes and the number of drops, made the response much slower.

   View case
4. IL-0526 OpenStreetMap.org went offline, then read-only, for 56 hours after HE.net routing equipment cut both fibre uplinks

   OpenStreetMap Foundation2024

   Symptoms

   At 04:01 UTC on 15 December 2024, StatusCake monitoring sent SMS alerts that OpenStreetMap.org was unreachable. The website and the mapping API had gone completely offline at 03:53 UTC.

   Network unreachableService outage

   Trigger

   HE.net's routing equipment in Amsterdam failed at 03:53 UTC on 15 December 2024.

   Root cause

   Dependency failure

   Component

   ISP and transit networksDatabase replication

   Contributing factor

   Single point of failureNo runbook

   Detection

   StatusCake monitoring detected the outage at 04:01 UTC and sent SMS alerts, and the incident was reported to the operations team at once.

   Remediation

   • Activated a read-only backup instance of OpenStreetMap.org in Dublin
   • Brought a new Equinix Internet link into service in Amsterdam, bypassing HE.net and restoring full functionality
   • HE.net replaced its routing hardware and restored its service, and routing through HE.net resumed

   Action items

   • Move to a multi-ISP architecture so another provider can take over when one fails
   • Research backup ISPs for the Amsterdam servers, including BGP and keeping HE.net as a fallback
   • Add deeper route and latency monitoring and an automated escalation protocol
   • Hold failover drills for switching between ISPs and keep replication up to date
   • Agree SLAs with HE.net and Equinix Internet with defined escalation procedures and response times
   • Prepare the 4G out-of-band system as a manual fallback uplink that can sync Postgres replication data, and upgrade its hardware
   • Document outage communication procedures and delegate communication during outages
   • Investigate decoupling the OAuth service so authentication keeps working while the database is read-only

   Lessons

   Relying on HE.net alone left the Amsterdam servers with a single point of failure, so OpenStreetMap will keep multiple upstream providers. The 4G out-of-band link proved invaluable and should be able to carry data synchronisation. Resumed replication would have restored editing faster than the read-only backup. Frequent contact with the ISP gave no useful estimates, so the SLA needs to be better understood. Alerting on upstream connectivity needs refining, and a read-only OpenStreetMap.org also blocked logins to services that were not affected, because it could not save new tokens.

   View case
5. IL-0347 About 25% of Scaleway FR-PAR-1 hypervisors lost VPC connectivity when both VPC BGP route reflectors got stuck

   Scaleway2024

   Symptoms

   On 18 October 2024, from 7:50 to 8:01 UTC, about 25% of the hypervisors in Scaleway's FR-PAR-1 availability zone lost VPC connectivity with the rest of the region, and products that use VPC failed with them. Public internet connectivity and Elastic Metal VPC connectivity were not affected.

   Network unreachablePacket lossNode NotReady

   Trigger

   At 6:20 UTC a Top of Rack switch in FR-PAR-1 had a software crash. One of the two BGP route reflectors for VPC ran on a hypervisor in that rack and was left in an abnormal state.

   Root cause

   Software defect

   Component

   Network control plane (SDN)BGP routingNetwork switchDatabaseManaged Kubernetes control plane and provider API

   Contributing factor

   Excessive blast radiusAutomation amplification

   Detection

   Health-check monitoring on both route reflectors detected the anomaly, and the Scaleway Customer Excellence and SRE teams detected the customer impact and opened an incident.

   Remediation

   • Fixed both VPC route reflectors by manual action

   Action items

   • Fixed the autohealing mechanism that had failed to fix route reflectors stuck in a stale state; the fix is https://github.com/FRRouting/frr/pull/17163
   • Run different software versions on the route reflectors so a single bug cannot stop several instances
   • Investigate the software issue that left the route reflectors in a stale state

   View case

By root cause10