IL-0520 cloud.gov · 2016-07-05databaseobservability
logs.cloud.gov trapped users in a login redirect loop because its full Redis session store stopped saving sessions
logs.cloud.gov is the log search service of cloud.gov. Its session management saves user login sessions to a Redis database, and the same Redis pushes data on to an Elasticsearch cluster. In July 2016 the Elasticsearch cluster was failing because of insufficient memory limits and because of a configuration error introduced when hardening scripts were applied to it. Redis could not push its data to Elasticsearch and ran out of space, its capacity hard-coded to use 1GB instead of 86% of available memory, so it stopped saving login sessions, and from 7:12pm Eastern Time on 5 July users of logs.cloud.gov were caught in a login redirect loop.
The team reconfigured Redis on the logsearch service to use a much larger memory instance, through cg-deploy-logsearch, so that logs were kept while the Elasticsearch cluster was down. It corrected the Elasticsearch memory setting and the hardening problem, which had set /tmp to noexec so that the JVM could not JIT jars and the instances would not restart, and redeployed the cluster. A disk encryption step made the redeployment slow. At 8:00pm Eastern Time on 6 July the cluster deployment was complete; the team redeployed the Kibana instance, checked the logs and logs.cloud.gov was restored.
- Severity
- –
- Scope
- Users could not log in to the logs.cloud.gov log search service from 7:12pm Eastern Time on 5 July 2016 until it was restored after 8:00pm Eastern Time on 6 July
- Users affected
- –
- Region
- –
- Time to detect
- –
- Time to resolve
- –
Symptoms
At 7:12pm Eastern Time on 5 July 2016, logs.cloud.gov, the cloud.gov log search service, sent users into a login redirect loop, so they could not log in to the service.
The loop came from the service's session management, which failed when it tried to save the user's login session data to the Redis database. Redis was not saving the sessions because it had run out of space: it could not push its data on to the Elasticsearch cluster, and the Elasticsearch cluster was failing too.
Recovery ran into two more problems. After the Elasticsearch configuration was updated and the cluster redeployed, the instances would not restart, because a hardening setting had made the /tmp directory noexec and the JVM could not JIT jars. Once the instances restarted, the disk encryption step made the redeployment slow, and the cluster deployment completed only at 8:00pm Eastern Time on 6 July.
Symptom Service outageWrites rejected
Root cause
Root cause Capacity exhaustion
- Trigger
- –
Detection
–
Remediation
- Moved Redis on the logsearch service to a much larger memory instance so it could keep logs while Elasticsearch was down
- Corrected the Elasticsearch memory limits and the /tmp noexec hardening setting and redeployed the cluster
- Redeployed the Kibana instance and verified the logs once the cluster deployment was complete
Action items
- Add a failure alert that checks whether logs are coming through
- Make Redis auto-scaling or highly available so that failures in more fragile downstream components do not cause data loss
- Extend the incident response plan to start with events that are not security-specific
- Create a chaos bot for the logs.cloud.gov service
Lessons
–
Sources
Related cases
IL-0432AutoReserve notifications silently went unsent as Memorystore for Redis 3.2 rejected writes with OOM below its limitHello Inc. · 2026same root cause · shared In-memory cache
IL-0575Amazon OpenSearch cluster went red under high heap usage and the admin screen's customer list would not openSonicmoov · 2026same root cause · shared Elasticsearch or OpenSearch
IL-0290GitLab.com's Redis Sidekiq cluster ran out of memory and lost quorum, causing 500 errors and a 12 million job backlogGitLab · 2026 · sev1same root cause · shared In-memory cache
IL-058129CM product display servers lost pods and saturated Netty event loops on the first day of the Igu Week saleMUSINSA · 2025same root cause · shared In-memory cache
IL-0100PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queriesPostHog · 2025same root cause · shared In-memory cache
IL-0067Redis filled its persistent volume, corrupted its append-only file and crash-looped on every restartBC Government · 2025 · sev1same root cause · shared In-memory cache
IL-0544Wasabi's database cache stopped responding when its queueing service filled up, and S3 failed in all regionsWasabi · 2024same root cause · shared In-memory cache
IL-0042Application logs stopped reaching Elasticsearch and OpenSearch for hours and were lost when OpenSearch ran out of disk spaceUK Ministry of Justice · 2024same root cause · shared Elasticsearch or OpenSearch
IL-0202A burst of queries and database writes exhausted connection pools and memory on Honeycomb's ingest hosts, and Kubernetes crash-loop back-off kept failed hosts offline so ingest capacity stayed too low to recoverHoneycomb · 2023 · sev1same root cause · shared In-memory cache
IL-0386After a move to Google Cloud, Postgres connection timeouts and memcached i/o timeouts spiked on GKE nodesincident.io · 2023same root cause · shared In-memory cache
IL-0453Hakuna API latency spiked every day during the ElastiCache Redis backup as memory filled and the backup went forklessHyperconnect · 2022same root cause · shared In-memory cache
IL-0212Many Slack users could not connect because a Vitess keyspace for channel membership was overloaded by cache-miss queriesSlack · 2022same root cause · shared In-memory cache
IL-0297Mezmo users could not sign in and log indexing lagged after the auth cache's Redis servers were OOM killedMezmo · 2019same root cause · shared In-memory cache
IL-0296Transloadit's us-east Redis cluster lost queued jobs and heartbeats, raising WORKER_JOB_ERROR errors for 81 customersTransloadit · 2016same root cause · shared In-memory cache
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0520
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0520/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.