Incident Library
Applied filters SymptomThroughput degradationFactorDependence on a warm cacheClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0297 Mezmo users could not sign in and log indexing lagged after the auth cache's Redis servers were OOM killed
    Mezmo2019
    Symptoms

    On 15 May 2019, alerts and customer reports showed that the Mezmo web application did not let users sign in, and the status page reported an indexing delay: incoming logs reached live tail and search late.

    Service outageOOMKilledDatabase overloadThroughput degradation
    Trigger

    A traffic spike that the tool tracking batches of logs was not sized to handle.

    Root cause
    Capacity exhaustion
    Component
    In-memory cacheDatabase
    Contributing factor
    Dependence on a warm cache
    Detection

    Alerts and customer reports in the early morning hours triggered the escalation procedure and woke the on-call engineers.

    Remediation
    • Resized specific parts of the cluster to handle a much higher load
    Action items
    • Retool the batch tracking system to reduce its direct impact on the Redis clusters
    • Add observability instrumentation around the batch tracking feature set
    • Improve monitoring for Redis
    • Remove the bottlenecks in the pipeline that were involved in this incident
    Lessons

    Mezmo took from the incident that deep instrumentation, tracing and observability across the platform matter more as features scale up.

    View case
Check your cluster for this

This case involves In-memory cache and Database. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion1 case

Component
In-memory cacheDatabase
Contributing factor
Dependence on a warm cache
Organisation
Mezmo
  1. IL-0297 Mezmo users could not sign in and log indexing lagged after the auth cache's Redis servers were OOM killed
    Mezmo2019