Cases1 case

1 to 1 of 1

1. IL-0452 Azar users on one carrier's 3G IP pool in one country increasingly hit read timed out errors on a STOMP connection

   Hyperconnect2024

   Symptoms

   In January 2024 a few customer-support reports came in through Hyperconnect's internal CS channel: some Azar users in one country, called XY in the write-up, reported a problem, and the errors pointed to an internal component called A. Component A had not changed around that time, service metrics did not visibly worsen and the reports were very few, so the teams found no lead and went back to other work.

   Intermittent timeouts

   Trigger

   The team cautiously estimates that carrier Z was gradually rolling out a setting that adjusts the QoS pool depending on whether a TLS connection includes SNI, so the hourly error count kept growing.

   Root cause

   DNS or network fault

   Component

   ISP and transit networks

   Contributing factor

   Unreviewed defaultsMonitoring blind spot

   Detection

   Customer-support reports from users in country XY reached the internal CS channel in January 2024; the problem was taken up only after a report showed those reports rising steadily into March.

   Remediation

   • Shipped Android and iOS hotfixes that include SNI in the TLS connection to component A
   • Fixed the iOS bug in reporting timeout errors in the same hotfix

   Action items

   • Use mobile proxy services such as Oxylabs to test connectivity to core components from global networks automatically and collect metrics

   Lessons

   In the retrospective the SRE team named as a problem that it started looking at the issue only as CS reports piled up, at least three months after it first appeared. A global service can lose connectivity for some users because a local network changes unexpectedly, and such problems can badly hurt the experience in a major country.

   View case

By root cause1