IL-0538 Sakura Internet · 2012-01-05database

Sakura Cloud storage maintenance deleted 53 in-use customer disks, and the disks could not be restored from backup

Sakura Cloud was released on 15 November 2011, and its storage device had failures on and off from 9 December 2011 to the end of March 2012. One of the problems was performance: from early January, as more customers created disks, the number of shared file systems on the storage device grew, and the CLI that the cloud system uses to create and delete file systems became slow to respond and to finish commands. The number of file systems was one reason for the slowdown, so in the emergency maintenance on 5 January Sakura Internet started deleting disks that had not finished being created because of the load and disks whose contracts had been cancelled. The load was too high for the normal procedure, so the team wrote a batch command for this maintenance and used it to delete the unneeded file systems all at once.

The batch command was not reviewed or tested enough, and nobody noticed that its rule for which kinds of file systems to delete (ones that failed to be created, ones that were cancelled) was wrong. It deleted 53 disks that were in use. These were file systems that had been created in advance as stock to prevent storage performance degradation, and the command took them for disks that had not been created properly. A backup mechanism existed, but backups had become less frequent because of the storage load, and the deleted disks were new enough that their backups had not started yet, so they could not be restored. After this incident Sakura Internet simplified the process for deleting unneeded file systems and made it a rule that scripts without enough review and testing are not run. The storage problems as a whole led Sakura Internet to add a second storage device on 12 March 2012 and later to replace the storage with a device it developed itself.

Severity

–

Scope

53 Sakura Cloud disks that customers were using were deleted and could not be restored

Users affected

–

Region

–

Time to detect

–

Time to resolve

–