Incident Library

IL-0560 PubNub · 2026-08-25networking

Some PubNub messages in the US East point of presence were misrouted with no error and not delivered right away

During planned internal load testing, PubNub took a group of internal routing servers out of service. Because of a configuration flaw, their network addresses were not fully retired, and the publishing layer kept them in its cache. One of those addresses was later reassigned to an unrelated internal component that answered normally on the same protocol and port. The publishing layer saw what looked like a successful response and got no error, so messages sent to that address were routed incorrectly. At 12:10 UTC on 25 August 2026 internal monitoring alerted that a very small subset of messages in one availability zone of the US East point of presence might not have been delivered right away to subscribers in the same region; all messages were persisted normally in the PubNub Persistence service, so no message data was lost. After the affected servers were identified, a rolling restart cleared the outdated addresses at 13:25 UTC, and at 22:00 UTC the same day a fix that makes sure outdated internal routing addresses are retired correctly was deployed to all publishing servers worldwide.

Severity
–
Scope
Publish/Subscribe in one availability zone of the US East point of presence: a very small subset of messages was not immediately delivered to subscribers in the same region; no message data was lost
Users affected
No customers reported impact
Region
US East point of presence, one availability zone
Time to detect
–
Time to resolve
1h 15m

Symptoms

At 12:10 UTC on 25 August 2026, PubNub's internal monitoring alerted that a very small subset of messages in one availability zone of the US East point of presence might not have been delivered right away to subscribers in the same region. Publish and Subscribe calls returned no errors, even though some messages did not appear in the responses.

Messages were still persisted normally in the PubNub Persistence service, so no message data was lost, and no other region, availability zone or PubNub service was affected. When the issue was mitigated at 13:55 UTC, the status page said it was believed to be related to a monitoring failure; the later investigation found misrouted messages. Synthetic monitoring returned to normal and detected no missed messages.

Symptom Misrouted requestsInconsistent reads from a data store

Root cause

Root cause Configuration change

Component Service discovery
Contributing factor –
Trigger
During planned internal load testing, a group of internal routing servers was removed from service.

Detection

PubNub's internal monitoring alerted at 12:10 UTC; synthetic monitoring then confirmed that no more messages were being missed.

Remediation

Action items

Lessons

–

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Service discovery among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.