Cases2 cases

1 to 2 of 2

1. IL-0575 Amazon OpenSearch cluster went red under high heap usage and the admin screen's customer list would not open

   Sonicmoov2026

   Symptoms

   During a regular meeting, Sonicmoov's customer success team reported that the customer list in the admin screen would not open, and customer inquiries arrived soon after. The author judged that the whole service was affected.

   Service outageMemory pressure

   Root cause

   Capacity exhaustion

   Component

   Elasticsearch or OpenSearch

   Contributing factor

   No capacity planning

   Detection

   Customer success reported that the customer list in the admin screen would not open, customer inquiries followed, and the AWS console showed the OpenSearch cluster red.

   Remediation

   • Restarted the OpenSearch nodes, which cleared the outage temporarily in about 30 minutes
   • Created a new index with 9 shards, one per OpenSearch node, with no replicas and refresh turned off during the copy
   • Copied the old index into the new one with an asynchronous _reindex and compared document counts
   • Set number_of_replicas to 1 and refresh_interval back to 1s on the new index
   • Moved the customer_data alias used by the application to the new index
   • Ran a second _reindex to copy documents written to the old index after the first copy

   Lessons

   As the service scales, its non-functional parts need work, and the author wants to deal with them before they cause trouble.

   View case
2. IL-0520 logs.cloud.gov trapped users in a login redirect loop because its full Redis session store stopped saving sessions

   cloud.gov2016

   Symptoms

   At 7:12pm Eastern Time on 5 July 2016, logs.cloud.gov, the cloud.gov log search service, sent users into a login redirect loop, so they could not log in to the service.

   Service outageWrites rejected

   Root cause

   Capacity exhaustion

   Component

   In-memory cacheElasticsearch or OpenSearch

   Contributing factor

   Overlapping failuresMonitoring blind spot

   Remediation

   • Moved Redis on the logsearch service to a much larger memory instance so it could keep logs while Elasticsearch was down
   • Corrected the Elasticsearch memory limits and the /tmp noexec hardening setting and redeployed the cluster
   • Redeployed the Kibana instance and verified the logs once the cluster deployment was complete

   Action items

   • Add a failure alert that checks whether logs are coming through
   • Make Redis auto-scaling or highly available so that failures in more fragile downstream components do not cause data loss
   • Extend the incident response plan to start with events that are not security-specific
   • Create a chaos bot for the logs.cloud.gov service

   View case

By root cause1