Cases2 cases

1 to 2 of 2

1. IL-0377 Upstash QStash schedules in the US region were lost, revived or run twice after the old and new schedulers diverged

   Upstash2026

   Symptoms

   From 1 May 2026, some QStash schedules in the US region behaved wrongly: changes to schedules did not take effect, paused or deleted schedules came back, and newly created schedules did not run as expected. Some scheduled jobs were also executed twice.

   Split brainDuplicate delivery or execution

   Trigger

   On 1 May 2026 a bug made QStash in the US region send traffic back to the legacy scheduling path, which had been left in the codebase as a fallback after the move to the new scheduling infrastructure.

   Root cause

   Software defect

   Component

   Job scheduler service

   Contributing factor

   Partial rollout

   Remediation

   • Made the new scheduling system the active source of truth again
   • Reconciled the data between the legacy and new scheduling systems
   • Wrote the missing schedule changes back into the new infrastructure
   • Resolved conflicts to keep user data and schedule continuity
   • Restored some previously paused or deleted schedules to avoid permanent data loss

   Action items

   • Remove obsolete fallback execution paths once a transition is complete
   • Add automated safeguards and alerts for unexpected fallback behaviour
   • Improve consistency validation between the systems
   • Expand rollback and reconciliation testing

   View case
2. IL-0376 A maintenance update set off needless full syncs between Upstash Redis replicas in eu-west-1, slowing the databases

   Upstash2025

   Symptoms

   On 25 February 2025, regional Upstash Redis databases in AWS eu-west-1 slowed down after a maintenance update, with very high CPU load on databases that have a replica in that region. Latency stayed high until about 21:23 UTC, even after the affected databases were unthrottled at 15:06 UTC.

   High latencySustained high node CPU

   Trigger

   A maintenance update to the regional Upstash Redis databases in AWS eu-west-1.

   Root cause

   Software defect

   Component

   In-memory cacheDatabase replication

   Contributing factor

   Automation amplification

   Remediation

   • Added more resources to the regional eu-west-1 cluster
   • Unthrottled the affected databases at 15:06 UTC and enabled more throughput

   Action items

   • Deploy a fix that stops regional databases from starting unnecessary full synchronisations
   • Contact regional database users about migrating to Upstash Global, which does not have this issue

   View case

By root cause1