Incident Library
Applied filters Root causeSoftware defectOrganisationUpstashClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0377 Upstash QStash schedules in the US region were lost, revived or run twice after the old and new schedulers diverged
    Upstash2026
    Symptoms

    From 1 May 2026, some QStash schedules in the US region behaved wrongly: changes to schedules did not take effect, paused or deleted schedules came back, and newly created schedules did not run as expected. Some scheduled jobs were also executed twice.

    Split brainDuplicate delivery or execution
    Trigger

    On 1 May 2026 a bug made QStash in the US region send traffic back to the legacy scheduling path, which had been left in the codebase as a fallback after the move to the new scheduling infrastructure.

    Root cause
    Software defect
    Component
    Job scheduler service
    Contributing factor
    Partial rollout
    Remediation
    • Made the new scheduling system the active source of truth again
    • Reconciled the data between the legacy and new scheduling systems
    • Wrote the missing schedule changes back into the new infrastructure
    • Resolved conflicts to keep user data and schedule continuity
    • Restored some previously paused or deleted schedules to avoid permanent data loss
    Action items
    • Remove obsolete fallback execution paths once a transition is complete
    • Add automated safeguards and alerts for unexpected fallback behaviour
    • Improve consistency validation between the systems
    • Expand rollback and reconciliation testing
    View case
  2. IL-0376 A maintenance update set off needless full syncs between Upstash Redis replicas in eu-west-1, slowing the databases
    Upstash2025
    Symptoms

    On 25 February 2025, regional Upstash Redis databases in AWS eu-west-1 slowed down after a maintenance update, with very high CPU load on databases that have a replica in that region. Latency stayed high until about 21:23 UTC, even after the affected databases were unthrottled at 15:06 UTC.

    High latencySustained high node CPU
    Trigger

    A maintenance update to the regional Upstash Redis databases in AWS eu-west-1.

    Root cause
    Software defect
    Component
    In-memory cacheDatabase replication
    Contributing factor
    Automation amplification
    Remediation
    • Added more resources to the regional eu-west-1 cluster
    • Unthrottled the affected databases at 15:06 UTC and enabled more throughput
    Action items
    • Deploy a fix that stops regional databases from starting unnecessary full synchronisations
    • Contact regional database users about migrating to Upstash Global, which does not have this issue
    View case
Check your cluster for this

These 2 cases involve In-memory cache and Database replication. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Software defect2 cases

Component
In-memory cache1Database replication1Job scheduler service1
Contributing factor
Automation amplification1Partial rollout1
Organisation
Upstash2
  1. IL-0377 Upstash QStash schedules in the US region were lost, revived or run twice after the old and new schedulers diverged
    Upstash2026
  2. IL-0376 A maintenance update set off needless full syncs between Upstash Redis replicas in eu-west-1, slowing the databases
    Upstash2025