Cases2 cases

1 to 2 of 2

1. IL-0560 Some PubNub messages in the US East point of presence were misrouted with no error and not delivered right away

   PubNub2026

   Symptoms

   At 12:10 UTC on 25 August 2026, PubNub's internal monitoring alerted that a very small subset of messages in one availability zone of the US East point of presence might not have been delivered right away to subscribers in the same region. Publish and Subscribe calls returned no errors, even though some messages did not appear in the responses.

   Misrouted requestsInconsistent reads from a data store

   Trigger

   During planned internal load testing, a group of internal routing servers was removed from service.

   Root cause

   Configuration change

   Component

   Service discovery

   Contributing factor

   –

   Detection

   PubNub's internal monitoring alerted at 12:10 UTC; synthetic monitoring then confirmed that no more messages were being missed.

   Remediation

   • Performed a rolling restart of the affected servers to clear the outdated routing addresses

   Action items

   • Deploy a fix to all publishing servers worldwide so that outdated internal routing addresses are retired correctly

   View case
2. IL-0371 Half of new connections to Render Postgres databases with point-in-time recovery failed while their backup jobs ran

   Render2023

   Symptoms

   After Point-in-time Recovery (PITR) was released to eligible new Render Postgres databases on 21 April 2023, 50% of new connections to PITR-enabled databases failed while a PITR backup job was running. A backup takes 15 to 45 minutes depending on database size, and existing connections, such as those held through connection pooling, kept working.

   Intermittent connection failuresMisrouted requests

   Trigger

   Point-in-time Recovery was released to eligible new databases on 21 April 2023, and its backup job carried an incorrect networking configuration.

   Root cause

   Faulty deployment

   Component

   DatabaseDatabase backups and snapshotsService discovery

   Contributing factor

   –

   Remediation

   • Disabled backups for affected databases until they could be scheduled in low-traffic hours
   • Changed the backup job's network labels so the routing layer no longer picks it up as a database

   Action items

   • Tighten the process for sharing incident updates so problems are not declared resolved too early
   • Use more specific network labels for the database routing layer so ineligible IPs are never added
   • Run a skipped asynchronous job in the next scheduled window instead of immediately
   • Use open source benchmarking software to reproduce issues customers notice

   View case

By root cause2