Cases6 cases

3 to 6 of 6

1. IL-0347 About 25% of Scaleway FR-PAR-1 hypervisors lost VPC connectivity when both VPC BGP route reflectors got stuck

   Scaleway2024

   Symptoms

   On 18 October 2024, from 7:50 to 8:01 UTC, about 25% of the hypervisors in Scaleway's FR-PAR-1 availability zone lost VPC connectivity with the rest of the region, and products that use VPC failed with them. Public internet connectivity and Elastic Metal VPC connectivity were not affected.

   Network unreachablePacket lossNode NotReady

   Trigger

   At 6:20 UTC a Top of Rack switch in FR-PAR-1 had a software crash. One of the two BGP route reflectors for VPC ran on a hypervisor in that rack and was left in an abnormal state.

   Root cause

   Software defect

   Component

   Network control plane (SDN)BGP routingNetwork switchDatabaseManaged Kubernetes control plane and provider API

   Contributing factor

   Excessive blast radiusAutomation amplification

   Detection

   Health-check monitoring on both route reflectors detected the anomaly, and the Scaleway Customer Excellence and SRE teams detected the customer impact and opened an incident.

   Remediation

   • Fixed both VPC route reflectors by manual action

   Action items

   • Fixed the autohealing mechanism that had failed to fix route reflectors stuck in a stale state; the fix is https://github.com/FRRouting/frr/pull/17163
   • Run different software versions on the route reflectors so a single bug cannot stop several instances
   • Investigate the software issue that left the route reflectors in a stale state

   View case
2. IL-0220 An overloaded AWS Transit Gateway dropped packets on 4 January 2021, and Slack's web tier saturated and went down

   Slack2021

   Symptoms

   During the Americas' morning of 4 January 2021, an external monitoring service paged Slack because error rates were creeping up. While the investigation was starting, Slack's own dashboarding and alerting service became unavailable, so responders had to work without their usual dashboards and alerts.

   High error rateMonitoring outagePacket lossService outageScale-up failureFile descriptor exhaustionCloud API rate limit exhausted

   Trigger

   On the first working day after the holidays, Slack's traffic went from the quietest time of the year to one of its biggest days overnight, with cold client caches pulling more data than usual, and one of Slack's AWS Transit Gateways did not scale fast enough for it.

   Root cause

   Cloud provider outage

   Component

   Transit gateway or network interconnectAutomated capacity manager

   Contributing factor

   Circular dependencyAutomation amplification

   Detection

   An external monitoring service paged Slack when error rates started creeping up during the Americas' morning. AWS engineers were separately alerted to the packet drops by AWS's internal monitoring.

   Remediation

   • Disabled automated downscaling of the web tier to keep serving capacity and allow investigation
   • Cleared up the broken, partly provisioned instances that filled the autoscaling groups
   • Repaired provision-service so new web tier instances could be provisioned and enter service
   • AWS engineers increased the Transit Gateway capacity manually across all Availability Zones

   Action items

   • AWS reviews its Transit Gateway scaling algorithms for large packet-per-second increases
   • Request a preemptive upscaling of the Transit Gateways at the end of the next holiday season
   • Run the dashboard service instances in the same VPC as their database to remove the Transit Gateway dependency
   • Load test provision-service regularly
   • Reevaluate health-checking and autoscaling configurations so they cannot overload provision-service again

   Lessons

   Monitoring has to stay as independent as possible of the infrastructure it watches, or it fails when it is needed most. Here the dashboard and alerting services failed because they ran in a different VPC from their databases and so depended on the Transit Gateways. provision-service had been load tested before, but this event exceeded the earlier scale-ups.

   View case
3. IL-0240 Cloudflare API requests failed and the dashboard slowed for hours after the authentication database lost its replicas

   Cloudflare2020

   Symptoms

   On 2 November 2020 the Cloudflare API and dashboard were degraded for six hours and 33 minutes. The success rate of API queries dropped at times to 75%, and the dashboard was up to 80 times slower than normal, while Cloudflare's edge network kept working.

   High error rateHigh latencyDatabase overloadWrites rejectedPacket loss

   Trigger

   At 14:43 UTC a network switch in a control plane data center partly failed: it kept LACP and BGP running but stopped forwarding some packets, and it recovered on its own six minutes later.

   Root cause

   DNS or network fault

   Component

   NIC, cable and switch port linkCoordination and key-value storeDatabase failover managerDatabaseDatabase replication

   Contributing factor

   Automation amplificationNo headroomHarmful manual action

   Detection

   At 14:43 UTC alerts fired because the misbehaving switch was unreachable to pings. Engineers then saw that the primary authentication database was overwhelmed.

   Remediation

   • Shed load from the primary authentication database by slowing SSL certificate pushes to the edge, emails and other features
   • Moved API traffic that could use read replicas to the secondary data center by hand
   • Sent authentication calls fully back to the primary data center when the dashboard got worse
   • Rebuilt the first database replica, which put itself back into service at 21:20 UTC and returned performance to normal
   • Ramped the turned-down services back up so asynchronous processing could catch up

   Action items

   • Remove the cluster management limitation that requires rebuilding all replicas after a primary is promoted
   • Redesign the user sessions system so dashboard traffic can be steered between data centers
   • Change the auto-remediation configuration so a database replica is not promoted so quickly
   • Tune each system involved to be more resilient to failures of its dependencies

   Lessons

   Every system in the chain had redundancy and none failed completely; each only degraded, which made the chain of events much harder to model and foresee. Automating replica promotion so that it could trigger on a minute's notice had been a point of pride, but for at least one database a fast promotion can do more harm than the failure it answers. Most general-purpose cluster managers use Raft or Paxos rather than Byzantine fault tolerant protocols, accepting a rare failure mode in exchange for a protocol that is simpler to implement correctly and debug.

   View case
4. IL-0537 Sakura Cloud host servers repeatedly lost their storage connection, about five minutes per port switchover

   Sakura Internet2012

   Symptoms

   From mid-December 2011, after Sakura Cloud launched, traffic between the host servers and the storage device grew, and the network connection between the hosts that carry users' servers and the storage device was cut again and again. Each cut caused by a port switchover lasted about five minutes.

   Storage disconnected or stale mountsPacket loss

   Trigger

   Traffic between the host servers and the storage device grew from mid-December 2011, after the service launched on 15 November 2011.

   Root cause

   Capacity exhaustion

   Component

   Cloud block storageInfiniBand fabric

   Contributing factor

   Automation amplificationUnrepresentative stagingLimited debug access

   Remediation

   • Switched the network interface monitoring from ICMP to a LINK state check, so the device fails over only on a physical link down
   • Changed the InfiniBand Subnet Manager's check from normal mode to heavy sweep mode
   • Updated the storage device firmware on 19 March, which stopped the packet loss
   • Added a second storage device on 12 March and migrated servers to it to reduce and spread the load

   Action items

   • Replace the current storage with an entirely different new storage device
   • Test the new storage in an environment equal to its maximum capacity under real load, with a free beta before paid service
   • Use network interface drivers and similar parts equivalent to those already proven on the host servers
   • Develop the new storage with in-house engineers so Sakura Internet can manage and operate it itself

   Lessons

   Sakura Internet wrote that it could not test the storage device at its performance limit, because it could not prepare as many servers as the design capacity and used load generated from predictions, so it could not respond quickly and accurately to problems in real operation. Packet loss on the InfiniBand port kept the device below its expected performance and took three months to fix by firmware, and its engineers could not grasp the whole of the device's specification and behaviour. Sakura Internet concluded that its process for choosing and validating the storage device had serious problems.

   View case

By root cause6