Cases2 cases

1 to 2 of 2

1. IL-0558 New PubNub instance types in FRA hit connectivity limits, and Pub/Sub, History and Presence saw intermittent errors

   PubNub2025

   Symptoms

   From 08:15 UTC on Friday 27 June 2025, PubNub's Pub/Sub, History and Presence services showed occasional, intermittent increases in latency and server errors in the Europe (FRA) region.

   High latencyHigh error rate

   Trigger

   PubNub introduced new instance types to its system to get access to more cloud server capacity across its regions.

   Root cause

   Capacity exhaustion

   Component

   Cloud load balancer

   Contributing factor

   Partial rolloutUnverified assumptionMonitoring blind spot

   Remediation

   • Configured the affected services to run on other instance types
   • Launched more capacity

   Action items

   • Make the new instance types emit metrics on OS thresholds and limits, so approaching or exceeding a limit is detected on any instance type
   • Scale ahead of demand and route traffic according to instance type

   View case
2. IL-0578 Static content behind an HAProxy load balancer in an OpenStack Octavia amphora was delayed by tens of seconds

   GMO Pepabo2020

   Symptoms

   In a GMO Pepabo production setup, delivery of static content was delayed by tens of seconds. The static content was served by nginx on upstream hosts behind HAProxy, which ran as the HTTP load balancer.

   High latency

   Root cause

   Capacity exhaustion

   Component

   HAProxyLinux kernelCloud load balancer

   Contributing factor

   Unreviewed defaults

   Remediation

   • Tuned net.ipv4.tcp_mem with sysctl on the HAProxy host, which ended the delays immediately

   Lessons

   On a VM with little RAM, the boot-time default of net.ipv4.tcp_mem can be low enough to push TCP into memory pressure. TCP out of memory is not the OOM killer, and net.ipv4.tcp_mem is a system-wide limit counted in pages, unlike the per-socket net.ipv4.tcp_rmem and net.ipv4.tcp_wmem counted in bytes. The values to watch are TCP mem in /proc/net/sockstat, press in /proc/net/protocols, and TCPMemoryPressures, TCPMemoryPressuresChrono and TCPAbortOnMemory in /proc/net/netstat.

   View case

By root cause1