Incident Library
Applied filters ComponentIn-memory cacheSymptomWrites rejectedOrganisationcloud.govClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0520 logs.cloud.gov trapped users in a login redirect loop because its full Redis session store stopped saving sessions
    cloud.gov2016
    Symptoms

    At 7:12pm Eastern Time on 5 July 2016, logs.cloud.gov, the cloud.gov log search service, sent users into a login redirect loop, so they could not log in to the service.

    Service outageWrites rejected
    Root cause
    Capacity exhaustion
    Component
    In-memory cacheElasticsearch or OpenSearch
    Contributing factor
    Overlapping failuresMonitoring blind spot
    Remediation
    • Moved Redis on the logsearch service to a much larger memory instance so it could keep logs while Elasticsearch was down
    • Corrected the Elasticsearch memory limits and the /tmp noexec hardening setting and redeployed the cluster
    • Redeployed the Kibana instance and verified the logs once the cluster deployment was complete
    Action items
    • Add a failure alert that checks whether logs are coming through
    • Make Redis auto-scaling or highly available so that failures in more fragile downstream components do not cause data loss
    • Extend the incident response plan to start with events that are not security-specific
    • Create a chaos bot for the logs.cloud.gov service
    View case
Check your cluster for this

This case involves In-memory cache and Elasticsearch or OpenSearch. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion1 case

Component
In-memory cacheElasticsearch or OpenSearch
Contributing factor
Monitoring blind spotOverlapping failures
Organisation
cloud.gov
  1. IL-0520 logs.cloud.gov trapped users in a login redirect loop because its full Redis session store stopped saving sessions
    cloud.gov2016