Incident Library
Applied filters Root causeFaulty deploymentSymptomService outageOrganisationDevsistersClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0567 Aurora MySQL behind Cookie Run: OvenBreak hit 100% CPU after a sales-ranking feature launch, and players could not play
    Devsisters2023
    Symptoms

    After further bug-fix deployments and reconnecting users on the launch day of the package sales ranking feature, a database alarm fired and the CPU of the Cookie Run: OvenBreak AWS Aurora MySQL database was at 100%. Database connections then failed, and players could not log in to or play the game.

    Database overloadService outage
    Trigger

    The package sales ranking feature was deployed, and further bug-fix deployments followed that reconnected users; each server update left more servers with empty caches that ran the ranking query against the database.

    Root cause
    Faulty deployment
    Component
    Database
    Contributing factor
    Inefficient queryDependence on a warm cacheUnrepresentative staging
    Detection

    A database alarm fired, and the CPU graph of the Aurora MySQL database showed 100%.

    Remediation
    • Put the game servers into maintenance
    • Added a switch for the package sales ranking feature and turned the feature off, which restored the service
    • Rewrote the ranking query to group products that share a sale start time and to drop the current-time condition
    • Moved the ranking query to a scheduler server that runs it periodically and stores one shared cached result
    • Turned the package sales ranking feature back on after both fixes
    Lessons

    A heavy query combined with per-server caches in a distributed server setup caused a problem nobody expected, so the points likely to cause trouble need to be found in advance. During an outage, restoring the service comes before finding the cause, and a switch that turns a feature off, built in ahead of time, is a safety device that helps run a stable service. The online ALTER itself went well with Percona Toolkit and was used again for later schema changes.

    View case
Check your cluster for this

This case involves Database. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Faulty deployment1 case

Component
Database
Contributing factor
Inefficient queryUnrepresentative stagingDependence on a warm cache
Organisation
Devsisters
  1. IL-0567 Aurora MySQL behind Cookie Run: OvenBreak hit 100% CPU after a sales-ranking feature launch, and players could not play
    Devsisters2023