Incident Library
Applied filters Root causeBreaking change in an upgradeFactorRetry stormClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases3 cases

1 to 3 of 3

  1. IL-0498 Clerk's Postgres database had load spikes every 15 minutes and requests failed and slowed intermittently for four days
    Clerk2025
    Symptoms

    At 13:08 UTC on 15 September 2025, Clerk's internal monitoring alerted on significant request failures and increased latency. Over the following days the request failures and latency spikes kept coming back while the Postgres database stayed unhealthy, with repeated spikes in database load.

    High error rateHigh latencyDatabase overloadConnection pool exhaustion
    Trigger

    At 05:30 UTC on 14 September 2025 the cloud provider automatically upgraded Clerk's Postgres database to a new minor version, which changed how Postgres handled connection locks.

    Root cause
    Breaking change in an upgrade
    Component
    DatabaseDatabase proxy and connection pooler
    Contributing factor
    Unpinned versionRetry stormMonitoring blind spotOverlapping failuresHarmful manual action
    Detection

    Internal monitoring alerted on significant request failures and increased latency at 13:08 UTC on 15 September. Shortly after the upgrade on 14 September it had already shown an increased but still acceptable database load.

    Remediation
    • Optimised several expensive queries, refined critical indexes and re-indexed the database
    • Blocked abusive traffic more aggressively for a time
    • Performed a manual minor version upgrade and increased database capacity
    • Added jitter to the connection pooling configuration so connections no longer recycled in step
    Action items
    • Evaluate other database providers that give more control over upgrades and downgrades
    • Isolate more services from each other, for example Sign Up and Sign In infrastructure from the rest of the system
    • Keep optimising the most costly database queries
    • Evaluate sharding and multi-tenant infrastructure to isolate one application from others
    Lessons

    Clerk's review found that connection metrics sampled every 60 seconds could not show a recycling pattern that lasted seconds and repeated every 15 minutes, that an attack and background jobs with similar periods pulled attention away from the database layer, that the Backend API looked healthy only because it manages far fewer connections, and that the release freeze removed the staggering of connection creation that normal deploys provide.

    View case
  2. IL-0314 Daily's newly upgraded Aurora MySQL database deadlocked on meeting-event queries and its call servers kept restarting
    Daily2023
    Symptoms

    At 9:47 AM Eastern (14:47 UTC) on 7 February 2023, Daily's database alerts reported a performance problem under normal operational load: a high level of lock contention on a database that had been upgraded the weekend before and had run normally since Monday. Users could get websocket connection errors when joining calls, and REST API requests could time out or return 5xx errors.

    Lock contention or deadlockHigh error rateIntermittent timeouts
    Trigger

    Database maintenance the weekend before had upgraded the database to a MySQL point release with a low-level behaviour change, and on 7 February, under normal load, that change produced the deadlocks Daily saw on Aurora MySQL 2.11.0.

    Root cause
    Breaking change in an upgrade
    Component
    Database
    Contributing factor
    Unrepresentative stagingAutomation amplificationRetry stormChange with a delayed effect
    Detection

    Database alerts at 9:47 AM Eastern reported a performance issue with high lock contention on the newly upgraded database.

    Remediation
    • Disabled MySQL binlog replication to the old database and restarted the database to reduce load
    • The database restarted by itself at 10:47 AM, the SFUs dropped the stale meeting updates without restarting, and the platform recovered
    • Rate-limited the POST /rooms endpoint to the same limit as DELETE /rooms/:name
    • Updated an index on the affected table in production on the Saturday evening to work around the MySQL behaviour change
    Action items
    • Work with AWS so the Aurora MySQL 2.11.0 deadlock issue is documented, understood and fixed in a future release
    • Add more API request patterns to the testing workload used for infrastructure changes
    • Add production monitoring alarms on more fine-grained database metrics
    Lessons

    Daily tests infrastructure changes under production-like workloads, but its synthetic workload did not have the shape that triggers these deadlocks. A deadlock behaviour change in MySQL 5.7 that was already known turned out far more severe in production than either Daily or the AWS Aurora team expected.

    View case
  3. IL-0294 CircleCI Docker and machine jobs stalled for nearly 12 hours after a VM cleanup queue lost all its RabbitMQ consumers
    CircleCI2021
    Symptoms

    From 13:30 UTC on 21 May 2021, CircleCI jobs on Docker, the Machine executor (including Windows, Mac, Arm and GPU) and remote Docker were delayed or blocked. Engineers had been paged at 13:00 for excessive delays in starting jobs, and the alerts showed CircleCI could not provision some types and classes of VM because it was hitting its CPU quotas in GCP.

    Throughput degradationCloud API rate limit exhaustedDatabase overload
    Trigger

    The upgrade of the RabbitMQ cluster from 3.8.9 to 3.8.16 on 20 May 2021, which brought in a 15 minute consumer acknowledgement timeout for all queue types.

    Root cause
    Breaking change in an upgrade
    Component
    Message brokerDatabase
    Contributing factor
    Change with a delayed effectIgnored early signalRetry stormExcessive blast radiusOverlapping failures
    Detection

    Engineers were paged at 13:00 UTC on 21 May for excessive delays in starting jobs. Earlier, a 02:31 notification that a VM destroyer queue had no consumers was handled by restarting pods, and a 10:11 alert that a VM destroy queue was backing up notified once at its threshold and did not page.

    Remediation
    • Blocked Linux machine jobs from entering the work queue
    • Had GCP raise the CPU quotas
    • Deleted old VMs by hand
    • Cancelled some workflows to free resources
    • Reset the VM services deployment so the VM destroy queues had consumers again
    • Scaled up the RDS database instance, at 18:20 and again at 19:15
    • Turned down VM provisioning and blocked remote-docker jobs to shed load
    • Scaled down all pods that created and destroyed VMs
    • Deployed a revised distribution backend that kept remote-docker jobs out of the provisioning queues
    • Brought provisioning and distribution back one component at a time and reversed the mitigation code changes
    Action items
    • Make VM provisioning more robust to changes in load so it can recover from thundering herds
    • Make VM provisioning independent so its failure does not affect Docker jobs
    • Review and improve observability of RabbitMQ and the production queues
    • Batch requests to avoid hitting GCP API rate limits
    • Improve incident response for long-running incidents
    View case
Check your cluster for this

These 3 cases involve Database and Database proxy and connection pooler. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Breaking change in an upgrade3 cases

Component
Database3Database proxy and connection pooler1Message broker1
Contributing factor
Retry storm3Change with a delayed effect2Overlapping failures2Automation amplification1Excessive blast radius1Ignored early signal1Harmful manual action1Monitoring blind spot1Unrepresentative staging1Unpinned version1
Organisation
CircleCI1Clerk1Daily1
  1. IL-0498 Clerk's Postgres database had load spikes every 15 minutes and requests failed and slowed intermittently for four days
    Clerk2025
  2. IL-0314 Daily's newly upgraded Aurora MySQL database deadlocked on meeting-event queries and its call servers kept restarting
    Daily2023
  3. IL-0294 CircleCI Docker and machine jobs stalled for nearly 12 hours after a VM cleanup queue lost all its RabbitMQ consumers
    CircleCI2021