IL-0537 Sakura Internet · 2012-06-25 (published)networkingdatabase
Sakura Cloud host servers repeatedly lost their storage connection, about five minutes per port switchover
Sakura Cloud, released on 15 November 2011, kept users' disks on a storage device connected to the host servers over an InfiniBand network. From early December the storage device's network interface lost packets intermittently as access grew. The interface's two-port redundancy relied on ICMP monitoring, which read the packet loss as a failure and switched to the standby port, and each switchover cut storage access for about five minutes. From late December a second problem appeared: the interface stopped answering the presence checks of the InfiniBand Subnet Manager, and the Subnet Manager cut the device off the network. These cuts were part of a series of storage failures that ran intermittently from the first failure on 9 December 2011 to the end of March 2012.
In early January Sakura Internet changed the interface monitoring from ICMP to a check of the LINK state, so the device switched ports only on a physical link down, which ended the failover problem, and changed the Subnet Manager's check from its normal mode, judged insufficient, to heavy sweep mode, which ended the Subnet Manager cut-offs. The packet loss itself continued until a firmware update of the storage device on 19 March. A second storage device was added on 12 March and servers that could move were migrated to it to spread the load, and with lower load on the first device no disk disconnection failures occurred from April onwards. In its review Sakura Internet wrote that the device had not been tested at its performance limit, that packet loss on the InfiniBand port took three months to fix by firmware, and that its engineers could not fully understand the device's specification and behaviour, so investigation with the manufacturer took a long time. Sakura Internet decided to replace the storage with a new device developed in-house.
- Severity
- –
- Scope
- Storage access of Sakura Cloud users' servers, cut for about five minutes at each interface switchover, and also at each Subnet Manager cut-off
- Users affected
- Sakura Cloud users
- Region
- –
- Time to detect
- –
- Time to resolve
- –
Symptoms
From mid-December 2011, after Sakura Cloud launched, traffic between the host servers and the storage device grew, and the network connection between the hosts that carry users' servers and the storage device was cut again and again. Each cut caused by a port switchover lasted about five minutes.
The storage device's network interface has two ports in a redundant pair. The active port was watched with ICMP (ping), and when ICMP replies stopped for a set period the port was treated as failed and the device switched to the standby port. From early December the interface lost packets intermittently, the monitor saw ICMP stop for the set period, and the device switched ports; the switchover itself cut communication with the hosts for about five minutes. Later investigation found that the interface was not faulty: the packet loss came from the growing volume of access.
From late December, storage access was also cut off several times for a different reason. The storage network runs on InfiniBand, where a Subnet Manager (SM) manages connections between devices. The storage device's network interface stopped answering the SM's presence checks, so the SM judged the device to be gone from the network and cut it off, although the interface showed no physical fault.
Root cause
Root cause Capacity exhaustion
- Trigger
- Traffic between the host servers and the storage device grew from mid-December 2011, after the service launched on 15 November 2011.
Detection
–
Remediation
- Switched the network interface monitoring from ICMP to a LINK state check, so the device fails over only on a physical link down
- Changed the InfiniBand Subnet Manager's check from normal mode to heavy sweep mode
- Updated the storage device firmware on 19 March, which stopped the packet loss
- Added a second storage device on 12 March and migrated servers to it to reduce and spread the load
Action items
- Replace the current storage with an entirely different new storage device
- Test the new storage in an environment equal to its maximum capacity under real load, with a free beta before paid service
- Use network interface drivers and similar parts equivalent to those already proven on the host servers
- Develop the new storage with in-house engineers so Sakura Internet can manage and operate it itself
Lessons
Sakura Internet wrote that it could not test the storage device at its performance limit, because it could not prepare as many servers as the design capacity and used load generated from predictions, so it could not respond quickly and accurately to problems in real operation. Packet loss on the InfiniBand port kept the device below its expected performance and took three months to fix by firmware, and its engineers could not grasp the whole of the device's specification and behaviour. Sakura Internet concluded that its process for choosing and validating the storage device had serious problems.
Sources
Related cases
IL-0350Hosts in an IONOS TXL cluster lost connectivity when an InfiniBand fabric resweep failed on a full multicast tableIONOS Cloud · 2026same root cause · shared InfiniBand fabric
IL-0249Large training jobs on a cluster of over 10,000 GPUs slowed for tens of seconds to nearly ten hours without crashingAlibaba Group · 2024same root cause · shared InfiniBand fabric
IL-0447An every RDS for MySQL instance slowed badly when its gp2 EBS BurstBalance ran out and IOPS fell to the baselineevery · 2024same root cause · shared Cloud block storage
IL-0321GCP snapshots on all Patroni nodes stalled GitLab.com's primary PostgreSQL disk, Patroni failed over and 5xx spikedGitLab · 2021same root cause · shared Cloud block storage
IL-0401Prow jobs on the k8s-prow-build GKE cluster failed and merges stalled as nodes hit IOPS throttling and kernel panicsKubernetes project · 2019same root cause · shared Cloud block storage
IL-0518gitlab.freedesktop.org on GKE ran short of volume space and a planned 30-minute storage upgrade became a 4-hour outagefreedesktop.org · 2018same root cause · shared Cloud block storage
IL-0468New JupyterHub student pods stuck Pending on MaxVolumeCount after a memory limit cut with node size unchangedUC Berkeley · 2017same root cause · shared Cloud block storage
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0537
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0537/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.