Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0295 Pusher's Channels publish API on the mt1 cluster was largely unavailable for 11 minutes while a Redis master failed over
    Pusher2022
    Symptoms

    At 10:42 UTC on 11 April 2022, the API for publishing messages to Pusher Channels on the mt1 cluster became largely unavailable, and from about 10:44 requests to the MT1 API returned more errors.

    Service outageHigh latencySplit brain
    Trigger

    An engineer restarted the master node of an mt1 Redis group to apply a larger replication buffer, after restarting both replicas of that group.

    Root cause
    Software defect
    Component
    In-memory cacheDatabase failover manager
    Contributing factor
    Unverified assumptionUnrepresentative staging
    Detection

    Pusher saw error rates rise for requests to the MT1 API from about 10:44 UTC.

    Action items
    • Fix the Redis shutdown script to check that failover has completed and the node is no longer a master before terminating
    View case
Check your cluster for this

This case involves In-memory cache and Database failover manager. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Software defect1 case

Component
In-memory cacheDatabase failover manager
Contributing factor
Unverified assumptionUnrepresentative staging
Organisation
Pusher
  1. IL-0295 Pusher's Channels publish API on the mt1 cluster was largely unavailable for 11 minutes while a Redis master failed over
    Pusher2022