Incident Library

Organisation

Sakura Internet

Sakura Internet · sakura-internet

2 cases · 2012

Root cause Capacity exhaustion1Operator error1
Component Cloud block storage2Database backups and snapshots1InfiniBand fabric1
Contributing factor Automation amplification1Untested backup1Limited debug access1Missed in review1Unrepresentative staging1
Symptom Data loss1Packet loss1Storage disconnected or stale mounts1

Cases

  1. IL-0537 Sakura Cloud host servers repeatedly lost their storage connection, about five minutes per port switchoverSakura Internet · 2012

    Root cause Capacity exhaustion

  2. IL-0538 Sakura Cloud storage maintenance deleted 53 in-use customer disks, and the disks could not be restored from backupSakura Internet · 2012

    Root cause Operator error