Cases2 cases

1 to 2 of 2

1. IL-0452 Azar users on one carrier's 3G IP pool in one country increasingly hit read timed out errors on a STOMP connection

   Hyperconnect2024

   Symptoms

   In January 2024 a few customer-support reports came in through Hyperconnect's internal CS channel: some Azar users in one country, called XY in the write-up, reported a problem, and the errors pointed to an internal component called A. Component A had not changed around that time, service metrics did not visibly worsen and the reports were very few, so the teams found no lead and went back to other work.

   Intermittent timeouts

   Trigger

   The team cautiously estimates that carrier Z was gradually rolling out a setting that adjusts the QoS pool depending on whether a TLS connection includes SNI, so the hourly error count kept growing.

   Root cause

   DNS or network fault

   Component

   ISP and transit networks

   Contributing factor

   Unreviewed defaultsMonitoring blind spot

   Detection

   Customer-support reports from users in country XY reached the internal CS channel in January 2024; the problem was taken up only after a report showed those reports rising steadily into March.

   Remediation

   • Shipped Android and iOS hotfixes that include SNI in the TLS connection to component A
   • Fixed the iOS bug in reporting timeout errors in the same hotfix

   Action items

   • Use mobile proxy services such as Oxylabs to test connectivity to core components from global networks automatically and collect metrics

   Lessons

   In the retrospective the SRE team named as a problem that it started looking at the issue only as CS reports piled up, at least three months after it first appeared. A global service can lose connectivity for some users because a local network changes unexpectedly, and such problems can badly hurt the experience in a major country.

   View case
2. IL-0259 Outbound packets from Fastmail did not reach part of the internet for about 19 hours, cutting off 3-5% of customers

   Fastmail2023

   Symptoms

   From about 03:00 UTC on 30 June 2023, Fastmail and a fraction of the internet could not communicate with each other, and Fastmail first heard of it from customers through support tickets and social media. The problem lasted about 19 hours and affected an estimated 3-5% of customers as well as some email servers.

   Network unreachable

   Root cause

   Dependency failure

   Component

   ISP and transit networks

   Contributing factor

   Single point of failureMonitoring blind spot

   Detection

   Customers told Fastmail through support tickets and social media. Fastmail's own monitors in multiple locations and its staff were not on affected networks and saw nothing.

   Action items

   • Select a provider for a second transit connection directly into Fastmail's servers, via Megaport or a provider present in 365's New Jersey datacenter, so that Fastmail controls the outbound path and can route around a network with problems

   Lessons

   With a single path for outbound traffic, Fastmail could not route around a provider's network problem and could not fix it itself. Being able to choose between several outbound paths would let Fastmail get everyone back online faster.

   View case

By root cause2