IL-0377 Upstash · 2026-05-01database
Upstash QStash schedules in the US region were lost, revived or run twice after the old and new schedulers diverged
As part of scalability work, Upstash had moved the QStash scheduling infrastructure in the US region to a new architecture, and the new system had become the source of truth for schedule state. A legacy execution path stayed in the codebase as a fallback. On 1 May 2026 a bug sent traffic back to the legacy path, which then accepted schedule updates on its own, separately from the new system. The two systems drifted apart, and users who had changed schedules since 24 April ended up with inconsistent schedule state. A limited number of users were affected, and most schedules ran normally throughout.
Upstash made the new scheduling system the active source of truth again, reconciled the data between the legacy and new systems, wrote the missing schedule changes into the new infrastructure and resolved conflicts so that user data and schedule continuity were kept. In some cases schedules that had been paused or deleted were restored to avoid losing data permanently. Main schedule functionality was back to normal on 5 May at 21:31 UTC, and the incident was marked resolved on 6 May at 08:05 UTC after Upstash checked that previously created schedules were delivered.
- Severity
- –
- Scope
- QStash schedules in the US region of users who created, edited, paused or deleted schedules between 24 April and 1 May 2026: updates not applied, paused schedules active again, deleted schedules back, new schedules not running, some jobs executed twice
- Users affected
- A limited number of users in the US region
- Region
- US region (US-EAST-1)
- Time to detect
- –
- Time to resolve
- –
Symptoms
From 1 May 2026, some QStash schedules in the US region behaved wrongly: changes to schedules did not take effect, paused or deleted schedules came back, and newly created schedules did not run as expected. Some scheduled jobs were also executed twice.
Only users who created, edited, paused or deleted schedules between 24 April and 1 May were affected; schedules nobody touched during that window kept running normally. On 5 May at 14:24 UTC the Upstash status page reported duplicate deliveries and said that schedules created between 24 April and 2 May 2026 were not running. Schedules created after the transition may have stopped running for a short time before they recovered.
Root cause
Root cause Software defect
- Trigger
- On 1 May 2026 a bug made QStash in the US region send traffic back to the legacy scheduling path, which had been left in the codebase as a fallback after the move to the new scheduling infrastructure.
Detection
–
Remediation
- Made the new scheduling system the active source of truth again
- Reconciled the data between the legacy and new scheduling systems
- Wrote the missing schedule changes back into the new infrastructure
- Resolved conflicts to keep user data and schedule continuity
- Restored some previously paused or deleted schedules to avoid permanent data loss
Action items
- Remove obsolete fallback execution paths once a transition is complete
- Add automated safeguards and alerts for unexpected fallback behaviour
- Improve consistency validation between the systems
- Expand rollback and reconciliation testing
Lessons
–
Sources
- https://status.upstash.com/incidents/9vxb0srq33b0 · official-status · archived copy
Related cases
–
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0377
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0377/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.