Cases1 case

1 to 1 of 1

1. IL-0579 Portfolio search latency on Amazon OpenSearch 1.3 spiked to about 1 second after migrating from Elasticsearch 6.8

   Classi2026

   Symptoms

   After Classi moved the search engine behind its Portfolio feature from Elasticsearch 6.8 to Amazon OpenSearch Service 1.3, the team saw in post-migration metrics that search responses were sometimes much slower. Most searches returned normally, but latency jumped at a certain frequency.

   High latency

   Trigger

   The migration from Elasticsearch 6.8 to OpenSearch 1.3, with indices that had no explicit refresh_interval.

   Root cause

   Breaking change in an upgrade

   Component

   Elasticsearch or OpenSearch

   Contributing factor

   Unreviewed defaultsUnverified assumption

   Detection

   The team noticed latency spikes in metrics checked after the migration and compared the maximum search latency before and after the migration in Datadog APM.

   Remediation

   • Set refresh_interval explicitly to 3s, which sharply reduced the latency spikes
   • Set refresh_interval explicitly to 1s in production

   Lessons

   The took value in OpenSearch's responses was already large, which could have narrowed the investigation to OpenSearch sooner. The slow log and the Profile API alone did not find the cause; asking when searches became slow, instead of where, led to the fix. The same setting can behave differently between versions, so defaults have to be checked as well.

   View case

By root cause1