IL-0295 Pusher · 2022-04-11database

Pusher's Channels publish API on the mt1 cluster was largely unavailable for 11 minutes while a Redis master failed over

Pusher runs Redis in groups of one master and two replicas, and since the beginning of November 2021 had been moving all groups to a new version of its Redis group module. Every node of the new groups runs a script at node termination that checks whether the node is a Redis master and, if it is, starts a failover to one of the replicas. To help the clusters cope with more traffic, Pusher increased the replication buffer size in the Redis configuration of every group, which needed each Redis node to be restarted; this had already been done on test clusters and on one smaller production cluster.

At 08:47 UTC an engineer began replacing nodes in the mt1 cluster and restarted both replica nodes of one Redis group, one at a time, without problems. At 10:42 the engineer did the same for the master. The failover started, but the Redis process was terminated before it completed. The shutdown script assumed that once the failover command had run successfully, the node could be terminated; on most clusters failover finishes before the Redis process is killed, but on a heavily loaded cluster such as mt1 the node terminated during failover and left the group in an inconsistent state that takes longer to recover from than no failover at all. After a period in which the old and new masters contended for leadership, the group recovered at 10:53 and the new master handled all messages.

Severity

–

Scope

Publishing to Channels on the mt1 cluster largely unavailable; messages delayed or lost

Users affected

–

Region

–

Time to detect

–

Time to resolve

11m