IL-0575 Sonicmoov · 2026-08-31 (published)databaseaws

Amazon OpenSearch cluster went red under high heap usage and the admin screen's customer list would not open

The service had scaled, and the amount of data in its customer index had grown with it. The old index had only 2 shards for about 80 million documents, so each shard held a large amount of data, and heap usage on the OpenSearch nodes rose sharply until the cluster went red. Restarting the OpenSearch nodes cleared the outage temporarily in about 30 minutes. Because the shortage of shards was likely to bring the outage back, the team then split the data over more shards.

The team created a new index whose name included the date, with 9 shards to match the 9 OpenSearch nodes, number_of_replicas 0 and refresh_interval "-1" during the copy. An asynchronous _reindex copied the old index into the new one, and the team compared document counts to confirm the copy. The team then set number_of_replicas to 1 and refresh_interval back to 1s, moved the customer_data alias that the application uses from the old index to the new one, and ran a second _reindex for the documents that had been written to the old index in the meantime. The person who had set up the infrastructure was no longer there, which made touching it daunting for the author.

Severity: –
Scope: The customer list in the admin screen could not be opened; the author describes the impact as service-wide
Users affected: –
Region: –
Time to detect: –
Time to resolve: –

Symptoms

During a regular meeting, Sonicmoov's customer success team reported that the customer list in the admin screen would not open, and customer inquiries arrived soon after. The author judged that the whole service was affected.

In the AWS console, the Amazon OpenSearch cluster behind the service showed red. Heap usage on the cluster had risen sharply.

Symptom Service outage Memory pressure

Root cause

Root cause Capacity exhaustion

Component Elasticsearch or OpenSearch

Contributing factor No capacity planning

Trigger: –

Detection

Customer success reported that the customer list in the admin screen would not open, customer inquiries followed, and the AWS console showed the OpenSearch cluster red.

Remediation

Restarted the OpenSearch nodes, which cleared the outage temporarily in about 30 minutes
Created a new index with 9 shards, one per OpenSearch node, with no replicas and refresh turned off during the copy
Copied the old index into the new one with an asynchronous _reindex and compared document counts
Set number_of_replicas to 1 and refresh_interval back to 1s on the new index
Moved the customer_data alias used by the application to the new index
Ran a second _reindex to copy documents written to the old index after the first copy

Action items

–

Lessons

As the service scales, its non-functional parts need work, and the author wants to deal with them before they cause trouble.

Sources

https://zenn.dev/sonicmoov/articles/db9cf58b3eb323 · blog

References

Related cases

IL-0042 Application logs stopped reaching Elasticsearch and OpenSearch for hours and were lost when OpenSearch ran out of disk spaceUK Ministry of Justice · 2024
same root cause · shared Elasticsearch or OpenSearch
IL-0520 logs.cloud.gov trapped users in a login redirect loop because its full Redis session store stopped saving sessionscloud.gov · 2016
same root cause · shared Elasticsearch or OpenSearch

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Elasticsearch or OpenSearch among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0575
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0575/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.