IL-0579 Classi · 2026-11-11 (published)databaseaws
Portfolio search latency on Amazon OpenSearch 1.3 spiked to about 1 second after migrating from Elasticsearch 6.8
Classi's Portfolio feature stores records that students make in daily school life, and Amazon OpenSearch Service is its search backend. The team moved more than 30 million documents by taking a snapshot of the Elasticsearch domain and restoring it into a new OpenSearch domain, and the application could switch between the two engines during the migration to compare results. The indices had no explicit refresh_interval, and the team believed the default of 1s applied.
In Elasticsearch 6.x and earlier, an index refreshes every second by default even without refresh_interval. Since Search Idle arrived in Elasticsearch 7.0, a shard with no explicit refresh_interval that receives no search for a while goes idle and stops refreshing in the background. OpenSearch is based on Elasticsearch 7.10 and keeps this behaviour, and index.search.idle.after defaults to 30s. When a search reaches an idle shard that needs a refresh, that first search waits for the refresh to finish, which produced the spikes after updates.
Setting refresh_interval explicitly to 3s cut the spikes sharply. Setting it explicitly back to 1s kept the improvement, which showed that what mattered was setting the value explicitly, which takes the shards out of Search Idle control, not the longer interval. The team set refresh_interval to 1s explicitly in production, and the intermittent latency spikes stopped.
- Severity
- –
- Scope
- Search for the Portfolio feature: maximum search latency intermittently about 1 second instead of about 100ms
- Users affected
- –
- Region
- –
- Time to detect
- –
- Time to resolve
- –
Symptoms
After Classi moved the search engine behind its Portfolio feature from Elasticsearch 6.8 to Amazon OpenSearch Service 1.3, the team saw in post-migration metrics that search responses were sometimes much slower. Most searches returned normally, but latency jumped at a certain frequency.
In Datadog APM, the maximum search latency had stayed around 100ms on Elasticsearch 6.8, while on OpenSearch 1.3 it intermittently reached about 1 second; normal latency had not become worse across the board. The migration had also replaced the Ruby client, elasticsearch-model and elasticsearch-rails, with opensearch-ruby, and the instance type m5.large.search (x86_64) with m7g.large.search (AWS Graviton, Arm), so the move to OpenSearch itself was not the only suspect.
The team first suspected slow queries. With the slow log threshold set to 0ms, the logged query times showed no delay matching what the application saw. Search API profile=true on one slow request showed "elapsed_ms": 867.5 in the application and "opensearch_took_ms": 839, while every shard's query time was under 1ms. Changing Keep-Alive, idle_timeout and the Faraday HTTP adapter and checking OpenSearch GC showed no clear link to the latency. Running the same query repeatedly with curl sometimes made only the first run slow, but not reproducibly.
The team then asked when searches became slow instead of where the time went, and found that a search right after documents were added or updated could be slow.
Symptom High latency
Root cause
Root cause Breaking change in an upgrade
- Trigger
- The migration from Elasticsearch 6.8 to OpenSearch 1.3, with indices that had no explicit refresh_interval.
Detection
The team noticed latency spikes in metrics checked after the migration and compared the maximum search latency before and after the migration in Datadog APM.
Remediation
- Set refresh_interval explicitly to 3s, which sharply reduced the latency spikes
- Set refresh_interval explicitly to 1s in production
Action items
–
Lessons
The took value in OpenSearch's responses was already large, which could have narrowed the investigation to OpenSearch sooner. The slow log and the Profile API alone did not find the cause; asking when searches became slow, instead of where, led to the fix. The same setting can behave differently between versions, so defaults have to be checked as well.
Sources
Related cases
–
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0579
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0579/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.