Cases1 case

1 to 1 of 1

1. IL-0575 Amazon OpenSearch cluster went red under high heap usage and the admin screen's customer list would not open

   Sonicmoov2026

   Symptoms

   During a regular meeting, Sonicmoov's customer success team reported that the customer list in the admin screen would not open, and customer inquiries arrived soon after. The author judged that the whole service was affected.

   Service outageMemory pressure

   Root cause

   Capacity exhaustion

   Component

   Elasticsearch or OpenSearch

   Contributing factor

   No capacity planning

   Detection

   Customer success reported that the customer list in the admin screen would not open, customer inquiries followed, and the AWS console showed the OpenSearch cluster red.

   Remediation

   • Restarted the OpenSearch nodes, which cleared the outage temporarily in about 30 minutes
   • Created a new index with 9 shards, one per OpenSearch node, with no replicas and refresh turned off during the copy
   • Copied the old index into the new one with an asynchronous _reindex and compared document counts
   • Set number_of_replicas to 1 and refresh_interval back to 1s on the new index
   • Moved the customer_data alias used by the application to the new index
   • Ran a second _reindex to copy documents written to the old index after the first copy

   Lessons

   As the service scales, its non-functional parts need work, and the author wants to deal with them before they cause trouble.

   View case

By root cause1