Incident Library

IL-0567 Devsisters · 2023-09-05 (published)databaseaws

Aurora MySQL behind Cookie Run: OvenBreak hit 100% CPU after a sales-ranking feature launch, and players could not play

The package sales ranking feature of Cookie Run: OvenBreak ranks the packages on sale by their sales over the last N hours, which needed the purchase time in the AWS Aurora MySQL table that holds every purchase since the game started. A plain ALTER TABLE to add a purchase_dt column with an index would have taken a table lock on a table tied directly to purchases, and the table was too large for the usual 2-hour maintenance. The in-place algorithm was tested on a copy of a test database: the first attempt failed with ERROR 1062 (23000): Duplicate entry '73817' for key 'PRIMARY' and the second succeeded, and since MySQL documentation describes this as an intermittent limit of the in-place algorithm, that plan was dropped. pt-online-schema-change from Percona Toolkit was then tested on a staging copy, on a production copy with chunk-size 250 (steady at about 45% CPU) and 500 (up to about 80% at first), and on staging under mocked user traffic, and was run in production with chunk-size 250 and a separate backup ready. The online ALTER finished without problems.

The outage came on the day the feature was released. The ranking query counted purchases with one OR branch per product, each with its own sale start time and a condition on the current time, and there was no index on (product_id, purchase_dt). Each game server is horizontally scaled in an AWS autoscaling group and keeps its own cache, and because planners upload new game data at any time, several game data versions coexist and each gets its own cache. The sale start time depends on the game data, so the query ran (number of servers) × (number of game data versions) times, and every server update left more servers with empty caches running it again. In the development environment there was little payment data and few servers, so the problem did not show.

The developer put the servers into maintenance, quickly added a switch to turn the ranking feature off and restored the service with it disabled. A composite index on (product_id, purchase_dt) improved the query by only about 10% in tests and was not added. The query was rewritten to group products that share a sale start time and to drop the meaningless current-time condition, which helped little but was applied. The main fix moved the query to a separate scheduler server: game servers read the result from a remote store and queue a request, and the scheduler server periodically runs the query once for the latest request and caches the result. With both changes the feature was turned back on and CPU stayed normal apart from small peaks when the scheduler ran the query.

Severity
–
Scope
Cookie Run: OvenBreak players could not log in to or play the game while database connections failed
Users affected
–
Region
–
Time to detect
–
Time to resolve
–

Symptoms

After further bug-fix deployments and reconnecting users on the launch day of the package sales ranking feature, a database alarm fired and the CPU of the Cookie Run: OvenBreak AWS Aurora MySQL database was at 100%. Database connections then failed, and players could not log in to or play the game.

Earlier that day, while the feature was being deployed, the CPU graph had already risen from its usual level of about 20% by about 10 to 20 points. The developer connected this to the query added for the new feature, but since it did not look serious yet, other problems found in features released at the same time were handled first.

Symptom Database overloadService outage

Root cause

Root cause Faulty deployment

Component Database
Contributing factor Inefficient queryDependence on a warm cacheUnrepresentative staging
Trigger
The package sales ranking feature was deployed, and further bug-fix deployments followed that reconnected users; each server update left more servers with empty caches that ran the ranking query against the database.

Detection

A database alarm fired, and the CPU graph of the Aurora MySQL database showed 100%.

Remediation

Action items

–

Lessons

A heavy query combined with per-server caches in a distributed server setup caused a problem nobody expected, so the points likely to cause trouble need to be found in advance. During an outage, restoring the service comes before finding the cause, and a switch that turns a feature off, built in ahead of time, is a safety device that helps run a stable service. The online ALTER itself went well with Percona Toolkit and was used again for later schema changes.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Database among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.