Cases1 case

1 to 1 of 1

1. IL-0496 Ably's Cassandra hit 100% CPU in every region and queries timed out globally, and the load returned on two more evenings

   Ably2020

   Symptoms

   Production alarms told Ably that Cassandra queries were timing out for a large fraction of queries globally. Multiple Cassandra instances in each region were pinned at 100% CPU, every production cluster was affected, and the disruption reached Ably's primary realtime messaging service. The first episode started at 21:22.

   Database overloadIntermittent timeouts

   Trigger

   A customer application with a large fleet of set-top box devices was programmed to re-register with Ably at the same time every day. A bug elsewhere in Ably's system meant that the rate limiting applied to every external API request was not applied to this request, and any query that took more than a few seconds was retried.

   Root cause

   Capacity exhaustion

   Component

   Database

   Contributing factor

   Retry stormUnverified assumption

   Detection

   Production alarms reported that Cassandra queries were timing out for a large fraction of queries globally.

   Remediation

   • Applied rate limiting to the request that had been reaching Cassandra without it
   • Redesigned the Cassandra layout and sharding strategy for that data so it no longer used counter columns

   Lessons

   Ably says theory and practice are different things at scale. Counter columns, which in theory should behave like a CRDT with the performance of Cassandra's regular storage, put two orders of magnitude more load on the instances, and further research showed they are not even eventually consistent. Besides the theoretical concerns, a design has to account for load, capacity, what happens when capacity is exceeded and how peer systems react when a service does not respond as expected. Ably's conclusion is not to use Cassandra counter columns.

   View case

By root cause1