IL-0558 PubNub · 2025-06-27networking

New PubNub instance types in FRA hit connectivity limits, and Pub/Sub, History and Presence saw intermittent errors

PubNub tunes each of its servers with many OS and kernel level settings. To get access to more cloud server capacity across its regions, PubNub recently began running its services on a more varied set of instance types. One setting that PubNub specified explicitly, which controls limits on network connectivity, was being silently overridden by PubNub's upstream load balancers, so the new instance types reached their connectivity limits. From 27 June 2025 this showed as intermittent latency and errors in Pub/Sub, History and Presence in the FRA region, and misleading early errors lengthened the investigation. Once the cause was found, on 30 June, the affected services were configured to run on other instance types and more capacity was launched.

Severity: –
Scope: Pub/Sub, History and Presence in the FRA region (European Points of Presence): intermittent increases in latency and errors
Users affected: –
Region: FRA (Europe)
Time to detect: –
Time to resolve: –

Symptoms

From 08:15 UTC on Friday 27 June 2025, PubNub's Pub/Sub, History and Presence services showed occasional, intermittent increases in latency and server errors in the Europe (FRA) region.

The errors seen first pointed the investigation in the wrong directions, so finding the cause took longer than PubNub aims for. A fix on 27 June was followed by monitoring, and the status page marked the incident resolved at 10:38 UTC that day, but the root cause was identified and corrected only on Monday 30 June.

Symptom High latency High error rate

Root cause

Root cause Capacity exhaustion

Component Cloud load balancer

Contributing factor Partial rollout Unverified assumption Monitoring blind spot

Trigger: PubNub introduced new instance types to its system to get access to more cloud server capacity across its regions.

Detection

–

Remediation

Configured the affected services to run on other instance types
Launched more capacity

Action items

Make the new instance types emit metrics on OS thresholds and limits, so approaching or exceeding a limit is detected on any instance type
Scale ahead of demand and route traffic according to instance type

Lessons

–

Sources

https://status.pubnub.com/incidents/20fn87h2hnyq · official-status · archived copy

Related cases

IL-0578 Static content behind an HAProxy load balancer in an OpenStack Octavia amphora was delayed by tens of secondsGMO Pepabo · 2020
same root cause · shared Cloud load balancer
IL-0333 A downsized RDS PostgreSQL database maxed out its CPU at US peak and failing ELB health checks took buildkite.com downBuildkite · 2016
same root cause · shared Cloud load balancer

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Cloud load balancer among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0558
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0558/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.