Incident Library
Applied filters ComponentIn-memory cacheFactorDependence on a warm cacheClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases6 cases

6 to 6 of 6

  1. IL-0297 Mezmo users could not sign in and log indexing lagged after the auth cache's Redis servers were OOM killed
    Mezmo2019
    Symptoms

    On 15 May 2019, alerts and customer reports showed that the Mezmo web application did not let users sign in, and the status page reported an indexing delay: incoming logs reached live tail and search late.

    Service outageOOMKilledDatabase overloadThroughput degradation
    Trigger

    A traffic spike that the tool tracking batches of logs was not sized to handle.

    Root cause
    Capacity exhaustion
    Component
    In-memory cacheDatabase
    Contributing factor
    Dependence on a warm cache
    Detection

    Alerts and customer reports in the early morning hours triggered the escalation procedure and woke the on-call engineers.

    Remediation
    • Resized specific parts of the cluster to handle a much higher load
    Action items
    • Retool the batch tracking system to reduce its direct impact on the Redis clusters
    • Add observability instrumentation around the batch tracking feature set
    • Improve monitoring for Redis
    • Remove the bottlenecks in the pipeline that were involved in this incident
    Lessons

    Mezmo took from the incident that deep instrumentation, tracing and observability across the platform matter more as features scale up.

    View case
Check your cluster for this

These 6 cases involve In-memory cache and Database. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause4

Dependency failure1 case

Component
In-memory cacheDatabase
Contributing factor
Overlapping failuresDependence on a warm cache
Organisation
OpenRouter
  1. IL-0408 OpenRouter's API returned 500s and false 401 "User not found" errors when its API key cache dropped and came back empty
    OpenRouter2026

Faulty deployment1 case

Component
In-memory cacheCI systemConfigMaps and SecretsDatabaseHelm
Contributing factor
Signal buried in noiseUnverified assumptionDependence on a warm cache
Organisation
Wikimedia Foundation
  1. IL-0324 Three MediaWiki deployments lost Memcached after a PHP 8.1 rollout and overloaded the External Storage database cluster
    Wikimedia Foundation2025

Software defect2 cases

Component
In-memory cache2Database1Elasticsearch or OpenSearch1
Contributing factor
Unverified assumption2Dependence on a warm cache2Ignored early signal1Harmful manual action1No runbook1
Organisation
Honeycomb1Wikimedia Foundation1
  1. IL-0216 Wikimedia on-wiki search was slow or rejected after a datacenter switchover overloaded one Elasticsearch node in codfw
    Wikimedia Foundation2023
  2. IL-0217 Honeycomb's main MySQL database locked up under cache-miss reads and ran out of connections, a 68-minute total outage
    Honeycomb2023

Capacity exhaustion2 cases

Component
In-memory cache2Database2Service discovery1
Contributing factor
Dependence on a warm cache2Automation amplification1Change during peak hours1Inefficient query1Retry storm1
Organisation
Mezmo1Slack1
  1. IL-0212 Many Slack users could not connect because a Vitess keyspace for channel membership was overloaded by cache-miss queries
    Slack2022
  2. IL-0297 Mezmo users could not sign in and log indexing lagged after the auth cache's Redis servers were OOM killed
    Mezmo2019