Incident Library

IL-0496 Ably · 2020-06-11 (published)database

Ably's Cassandra hit 100% CPU in every region and queries timed out globally, and the load returned on two more evenings

Ably uses a globally distributed and replicated Cassandra cluster for persistent storage of messages and some message metadata, and also for other entities. About two years before the incidents, Ably adopted Cassandra counter columns to get an O(1) count of certain partitions, as part of a sharding strategy for records about push device registrations; counting the cells of a partition with a query is O(N). Counter columns can only exist in tables made entirely of counter columns, a counter table can never be deleted, and counter updates are not idempotent. Ably accepted these limits because it only needed approximate counts, and expected counters, which can be built as a CRDT, to perform like Cassandra's regular storage model.

In early 2020, over three evenings, Cassandra instances in every region were pinned at 100% CPU and queries timed out; the first two episodes ended by themselves after about 10 minutes. Scaling the cluster horizontally and vertically did not stop the spikes. Instrumentation added after the second episode showed a huge number of requests for one operation on one customer account, whose fleet of set-top boxes re-registered at the same time every day. Each request made several Cassandra upserts and one update to a counter table. Any query that took more than a few seconds was retried, which turned into many millions of requests once load crossed a threshold, and a bug elsewhere in Ably's system let this request skip the rate limiting that applies to every external API request. Applying rate limiting stopped the disruption.

Ably then looked at why the load before the retries had degraded Cassandra so much. The team suspected the counter column, because other requests that did not touch it had never caused comparable load, even at much higher rates. Ably redesigned the Cassandra layout for that data and changed the sharding strategy so that it used no counter columns, and deployed the change a few days later. Removing the counter column operation cut the load on the Cassandra instances by two orders of magnitude.

Severity
–
Scope
Ably's primary realtime messaging service: operations that need a Cassandra lookup, such as the first use of an application in a given location and the persistence of some access tokens, failed
Users affected
some customers were significantly affected
Region
all regions
Time to detect
–
Time to resolve
–

Symptoms

Production alarms told Ably that Cassandra queries were timing out for a large fraction of queries globally. Multiple Cassandra instances in each region were pinned at 100% CPU, every production cluster was affected, and the disruption reached Ably's primary realtime messaging service. The first episode started at 21:22.

Most message processing at Ably does not touch Cassandra, but some operations do database lookups, for example the first use of an application in a given location and the persistence of some access tokens. These operations failed, and some customers were significantly affected. The statistics on inbound requests showed no spike in any single request type, so it was not even clear that external load was the cause.

Redirecting traffic would not help, because every region and every cluster was affected, and Cassandra clusters cannot be scaled in real time. Just as the team was about to scale Cassandra anyway, the load subsided at 21:32 and service returned to normal. Over the next day the team ran updates and repairs and went through statistics without finding a problem, then scaled the cluster horizontally and vertically. Exactly 24 hours after the first episode the same 100% CPU spike came back in all regions with no visible demand spike, and again subsided after 10 minutes.

In the following 24 hours the team added instrumentation to the API layer that sits on top of Cassandra, prepared to capture DTrace logs from affected instances, and scaled vertically again to the largest instance and volume sizes available. When the load came back at 21:22, the cluster did not quite max out, and the new data showed a huge number of requests for a single operation on a single customer account.

Symptom Database overloadIntermittent timeouts

Root cause

Root cause Capacity exhaustion

Component Database
Contributing factor Retry stormUnverified assumption
Trigger
A customer application with a large fleet of set-top box devices was programmed to re-register with Ably at the same time every day. A bug elsewhere in Ably's system meant that the rate limiting applied to every external API request was not applied to this request, and any query that took more than a few seconds was retried.

Detection

Production alarms reported that Cassandra queries were timing out for a large fraction of queries globally.

Remediation

Action items

–

Lessons

Ably says theory and practice are different things at scale. Counter columns, which in theory should behave like a CRDT with the performance of Cassandra's regular storage, put two orders of magnitude more load on the instances, and further research showed they are not even eventually consistent. Besides the theoretical concerns, a design has to account for load, capacity, what happens when capacity is exceeded and how peer systems react when a service does not respond as expected. Ably's conclusion is not to use Cassandra counter columns.

Sources

References

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Database among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.