Cases1 case

1 to 1 of 1

1. IL-0520 logs.cloud.gov trapped users in a login redirect loop because its full Redis session store stopped saving sessions

   cloud.gov2016

   Symptoms

   At 7:12pm Eastern Time on 5 July 2016, logs.cloud.gov, the cloud.gov log search service, sent users into a login redirect loop, so they could not log in to the service.

   Service outageWrites rejected

   Root cause

   Capacity exhaustion

   Component

   In-memory cacheElasticsearch or OpenSearch

   Contributing factor

   Overlapping failuresMonitoring blind spot

   Remediation

   • Moved Redis on the logsearch service to a much larger memory instance so it could keep logs while Elasticsearch was down
   • Corrected the Elasticsearch memory limits and the /tmp noexec hardening setting and redeployed the cluster
   • Redeployed the Kibana instance and verified the logs once the cluster deployment was complete

   Action items

   • Add a failure alert that checks whether logs are coming through
   • Make Redis auto-scaling or highly available so that failures in more fragile downstream components do not cause data loss
   • Extend the incident response plan to start with events that are not security-specific
   • Create a chaos bot for the logs.cloud.gov service

   View case

By root cause1