Cases1 case

1 to 1 of 1

1. IL-0297 Mezmo users could not sign in and log indexing lagged after the auth cache's Redis servers were OOM killed

   Mezmo2019

   Symptoms

   On 15 May 2019, alerts and customer reports showed that the Mezmo web application did not let users sign in, and the status page reported an indexing delay: incoming logs reached live tail and search late.

   Service outageOOMKilledDatabase overloadThroughput degradation

   Trigger

   A traffic spike that the tool tracking batches of logs was not sized to handle.

   Root cause

   Capacity exhaustion

   Component

   In-memory cacheDatabase

   Contributing factor

   Dependence on a warm cache

   Detection

   Alerts and customer reports in the early morning hours triggered the escalation procedure and woke the on-call engineers.

   Remediation

   • Resized specific parts of the cluster to handle a much higher load

   Action items

   • Retool the batch tracking system to reduce its direct impact on the Redis clusters
   • Add observability instrumentation around the batch tracking feature set
   • Improve monitoring for Redis
   • Remove the bottlenecks in the pipeline that were involved in this incident

   Lessons

   Mezmo took from the incident that deep instrumentation, tracing and observability across the platform matter more as features scale up.

   View case

By root cause1