Incident Library

IL-0452 Hyperconnect · 2024-06-24 (published)networking

Azar users on one carrier's 3G IP pool in one country increasingly hit read timed out errors on a STOMP connection

Hyperconnect's Azar app connects to component A over the STOMP protocol. The team's existing tool for monitoring regional network connectivity ran from data centres and supported only a limited set of protocols, so it could not test STOMP from mobile networks, and the team had no way to track problems in part of one local network. The SRE team therefore used Oxylabs, a SaaS that provides mobile SOCKS5 proxies by country, and before each connect request fetched https://ifconfig.co/json to learn which carrier the proxy device was on, since the carrier could not be chosen.

The first clue was that a plain curl request through an HTTP proxy got a quick answer even for the affected users. Comparing the reproducing script with earlier code that had failed to reproduce the problem showed the difference: whether the TLS Client Hello carried SNI. The library used for STOMP did not include SNI in TLS connections by default. This had not been a problem until recently; the team estimates that carrier Z gradually rolled out QoS settings based on SNI, and found that example configurations for software such as F5's Local Traffic Manager make it easy to drop packets without SNI or send them to a separate pool.

The Android and iOS teams shipped hotfixes that add SNI. Before release the team compared versions with and without SNI through SOCKS proxies, using the tun2socks app on Android and ShadowRocket on iOS. The iOS hotfix also fixed the bug in timeout error reporting. As adoption of the new versions rose, the errors fell gradually and the related CS reports stopped.

Severity
–
Scope
Azar users on carrier Z's 3G IP pool in country XY could not connect to component A over STOMP and got read timed out errors, many of them repeatedly
Users affected
As of 2024-03-26: 1,322 users with one error, 1,471 with 2 to 10, 798 with 10 to 100 and 147 with more than 100
Region
One country, called XY in the write-up
Time to detect
–
Time to resolve
–

Symptoms

In January 2024 a few customer-support reports came in through Hyperconnect's internal CS channel: some Azar users in one country, called XY in the write-up, reported a problem, and the errors pointed to an internal component called A. Component A had not changed around that time, service metrics did not visibly worsen and the reports were very few, so the teams found no lead and went back to other work.

Later a report showed that the related CS cases kept coming and rose linearly, from x at the end of December to the third week of March. In the client error logs, which Hyperconnect analyses in near real time with Databricks and an internal platform, about 20 of several hundred error types appeared more than 1,000 times a day. Only one of them, on Android, grew steadily: read timed out, which the Android app reports when its connection to component A is not made in time. Filtering for one carrier, called Z, showed the same rising trend at almost the same absolute numbers. As of 2024-03-26, 1,322 users had hit the error once, 1,471 two to ten times, 798 ten to 100 times and 147 more than 100 times, which pointed to repeated failures making Azar unusable for them. iOS showed no such pattern because of a bug in the logic that reports timeout errors.

Component A served traffic from every country, so the team ruled out the application, and the long duration ruled out an AWS outage. Routing country XY's traffic to component A through Global Accelerator, so that the local network connected straight onto AWS's network, did not change the error trend, which narrowed the problem to the local network. The affected users timed out only on STOMP connections to component A, not on all traffic.

To reproduce it, the SRE team connected Azar's golang STOMP test script through Oxylabs mobile SOCKS5 proxies in country XY. Only devices on a specific IP range of carrier Z, a 3G IP pool, got answers very late, after 40 seconds to 2 minutes, far beyond the connection timeout, and a user who failed once kept failing on retries.

Symptom Intermittent timeouts

Root cause

Root cause DNS or network fault

Component ISP and transit networks
Contributing factor Unreviewed defaultsMonitoring blind spot
Trigger
The team cautiously estimates that carrier Z was gradually rolling out a setting that adjusts the QoS pool depending on whether a TLS connection includes SNI, so the hourly error count kept growing.

Detection

Customer-support reports from users in country XY reached the internal CS channel in January 2024; the problem was taken up only after a report showed those reports rising steadily into March.

Remediation

Action items

Lessons

In the retrospective the SRE team named as a problem that it started looking at the issue only as CS reports piled up, at least three months after it first appeared. A global service can lose connectivity for some users because a local network changes unexpectedly, and such problems can badly hurt the experience in a major country.

Sources

References

Related cases

–

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, ISP and transit networks among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.