Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases7 cases

6 to 7 of 7

  1. IL-0520 logs.cloud.gov trapped users in a login redirect loop because its full Redis session store stopped saving sessions
    cloud.gov2016
    Symptoms

    At 7:12pm Eastern Time on 5 July 2016, logs.cloud.gov, the cloud.gov log search service, sent users into a login redirect loop, so they could not log in to the service.

    Service outageWrites rejected
    Root cause
    Capacity exhaustion
    Component
    In-memory cacheElasticsearch or OpenSearch
    Contributing factor
    Overlapping failuresMonitoring blind spot
    Remediation
    • Moved Redis on the logsearch service to a much larger memory instance so it could keep logs while Elasticsearch was down
    • Corrected the Elasticsearch memory limits and the /tmp noexec hardening setting and redeployed the cluster
    • Redeployed the Kibana instance and verified the logs once the cluster deployment was complete
    Action items
    • Add a failure alert that checks whether logs are coming through
    • Make Redis auto-scaling or highly available so that failures in more fragile downstream components do not cause data loss
    • Extend the incident response plan to start with events that are not security-specific
    • Create a chaos bot for the logs.cloud.gov service
    View case
  2. IL-0278 A weekly update put a Redis Cloud management process into an infinite loop, and most nodes of a GCP cluster failed
    Redis2016
    Symptoms

    On 8 February 2016 at 17:45 UTC, four nodes of the Redis Cloud cluster us-central1-1-1.gce.garantiadata.com on Google Cloud Platform stopped responding, four minutes after a first node had failed. The databases whose endpoints end in us-central1-1-1.gce.garantiadata.com went down.

    Service outageFailover failure
    Trigger

    The weekly upgrade of Redis Cloud's non-critical cluster components reached its first Google Cloud Platform cluster, us-central1-1-1.gce.garantiadata.com, at 17:30 UTC.

    Root cause
    Faulty deployment
    Component
    In-memory cacheDatabase failover managerAuthoritative DNS servers and records
    Contributing factor
    Overlapping failures
    Remediation
    • Shut down and terminated all nodes of the cluster
    • Deleted all of the cluster's endpoints from its DNS zone, removing by hand the entries left by throttled deletes
    • Provisioned new cluster nodes from scratch without the weekly update
    • Reconnected the original nodes' storage to the new nodes so persistence-enabled databases could be recovered
    • Bootstrapped and launched the databases, recovering the ones with persistence from storage
    Action items
    • Suspend all automatic weekly upgrades until a procedure that prevents this kind of outage is in place
    • Revalidate the QA processes for DevOps code deployments
    • Improve the tooling so it handles throttling of DNS requests
    View case
Check your cluster for this

These 7 cases involve In-memory cache and Database. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause5

Dependency failure1 case

Component
In-memory cacheDatabase
Contributing factor
Overlapping failuresDependence on a warm cache
Organisation
OpenRouter
  1. IL-0408 OpenRouter's API returned 500s and false 401 "User not found" errors when its API key cache dropped and came back empty
    OpenRouter2026

Capacity exhaustion2 cases

Component
In-memory cache2CNI plugin1Database1Elasticsearch or OpenSearch1
Contributing factor
Overlapping failures2Unreviewed defaults1Monitoring blind spot1
Organisation
cloud.gov1incident.io1
  1. IL-0386 After a move to Google Cloud, Postgres connection timeouts and memcached i/o timeouts spiked on GKE nodes
    incident.io2023
  2. IL-0520 logs.cloud.gov trapped users in a login redirect loop because its full Redis session store stopped saving sessions
    cloud.gov2016

Software defect2 cases

Component
In-memory cache2CDN and edge network1Coordination and key-value store1Database1Service discovery1
Contributing factor
Monitoring blind spot2Overlapping failures2Change with a delayed effect1Unpatched known bug1Harmful manual action1Single point of failure1
Organisation
Discord1Roblox1
  1. IL-0213 Roblox was down for 73 hours because key-value writes on its single Consul cluster slowed from under 300 ms to 2 seconds
    Roblox2021
  2. IL-0388 A Google Cloud live migration took a Discord Redis primary offline, and the failover broke API instances and then guilds
    Discord2017

Configuration change1 case

Component
In-memory cacheDatabase replicationDatabase failover manager
Contributing factor
Automation amplificationOverlapping failures
Organisation
Hatena
  1. IL-0430 Mackerel's Redis stopped answering during a replication setup, keepalived failed over by mistake, and Mackerel degraded
    Hatena2018

Faulty deployment1 case

Component
In-memory cacheAuthoritative DNS servers and recordsDatabase failover manager
Contributing factor
Overlapping failures
Organisation
Redis
  1. IL-0278 A weekly update put a Redis Cloud management process into an infinite loop, and most nodes of a GCP cluster failed
    Redis2016