Cases1 case

1 to 1 of 1

1. IL-0300 Basecamp 3 went read-only for almost five hours when its events table reached the ceiling of 2,147,483,647

   37signals2018

   Symptoms

   On November 9th, 2018, from 7:21am to 12:11pm CST, Basecamp 3 was in read-only mode. Users could open existing messages, to-do lists and files, but could not add anything new or change anything that was already there.

   Writes rejected

   Trigger

   The busy events table, which records almost every activity in Basecamp 3, reached its ceiling of 2,147,483,647, and new events could not be written.

   Root cause

   Capacity exhaustion

   Component

   Database

   Contributing factor

   No capacity planning

   Remediation

   • Raised the ceiling of the tables from 2,147,483,647 to 9,223,372,036,854,775,807, the same change as the Rails 5.1 default

   Lessons

   The outage was avoidable. The team did not account for how quickly the events table was running out of headroom while it was already working on the problem, and it should have reviewed its own database schema when Rails 5.1 announced the new default in 2017.

   View case

By root cause1