Incident Library

IL-0456 Inflab · 2022-01-03databaseaws

Inflearn went down six times during a January 2022 free-course event as slow queries piled up on Aurora PostgreSQL

Inflearn runs as a monolithic Node.js project on ECS Fargate, 10 to 30 instances depending on traffic, with Aurora PostgreSQL 10.16 as one writer and 2 to 7 readers. Most reads go to the readers and writes to the writer, but because services do not have their own databases, a problem in one part spreads to the whole service. The event, 125 courses worth 5,016,000 won in total offered for free, went far beyond expectations: more than 10,000 retweets, shares on many IT communities, traffic 5 to 20 times normal and 76,730 new sign-ups in 7 days. The first outages were handled with quick fixes: an index for the interested-companies query, a changed cache reset and regeneration interval, moving read queries off the writer, index-friendly rewrites or switching features off, dropping the course description column from select * queries, and splitting IN lists into batches of 20 IDs run with Promise.all.

The 17 January outage, slow queries on a table of under 10,000 rows, led the team to check tuple statistics on the writer DB. The course table had 99% dead tuples. Autovacuum should have been working on it, so the team ran VACUUM by hand and then watched the dead tuple rate, which climbed again very fast. The course table kept the number of students as a column, and every enrollment ran an UPDATE that added 1 to it. Because PostgreSQL's MVCC writes a new tuple on every update and leaves the old one as a dead tuple, and the course table's rows carried large columns such as the HTML course description, the frequent updates produced large dead tuples in bulk, made worse by the surge of students enrolling in 100 or more courses at once. That bottleneck slowed every query on the course table and was the root cause throughout the event.

The team chose to move the large course description column into a separate table rather than move the student count column, which would have meant working out its effect on the other existing features. After that, the event caused no further outages.

Severity
–
Scope
Inflearn was unusable at peak hours on 3, 4, 7 and 17 January 2022, six outages in total, including 1 h 30 min and 2 h on 3 to 4 January and 3 h on the morning of 4 January
Users affected
–
Region
–
Time to detect
–
Time to resolve
–

Symptoms

During a 100% discount event that made 125 courses free, Inflearn received 5 to 20 times its normal traffic, and on four days, 3, 4, 7 and 17 January 2022, the service became unusable around 9 a.m. and 10 p.m. There were six outages in all.

They showed as large numbers of slow queries on the Aurora PostgreSQL 10.16 database and, at times, writer CPU above 90%.

On 3 January, from 20:30 to 22:00, slow queries from the course pages' interested-companies lookup, part of which had no index, quickly used up the connections of the Node processes; later requests waited until the processes were killed, and although ECS added new processes, every other query those processes were running was cancelled, so the whole service stopped. From 23:30 to 01:30 the writer DB's CPU stayed above 90%: cache data moved from Redis into the database lived in a single large JSON row, cache resets and regeneration under the higher load hit that row at once and led to deadlocks, and read queries not yet moved to the readers added load.

On 4 January from 09:00 to 12:00, and again in the afternoon, course lookups were the slow queries, all of the top nine. Most could not use indexes, using lower("column") = lower(parameter) or negative conditions such as != and NOT IN, so tens to hundreds of full table scans ran at once, and select * pulled a 10 KB course description column of HTML into every course query. Scaling the database to 8 r5.8xlarge instances bought only a few tens of minutes. Finding the slow queries took under 30 minutes, but fixing them took 3 hours because every query was wrapped in functions, hooks and custom events, and the code had no types and no tests. On 7 January from 21:00 a course query with where id in (...) ran full table scans once the list passed 100 IDs, as students put all 125 courses in the cart and checked out at once.

Before the second event the team upgraded PostgreSQL from 10.16 to 11.13, set a 5-second query timeout, raised max connections and tuned queries over 2 seconds, and the busiest time of that week, 09:00 on Saturday 15 January, passed without trouble. At 22:00 on 17 January RDS metrics began to swing, and at 22:30 the sixth outage started, with slow queries even on a course table of fewer than 10,000 rows. Because every bottleneck so far had shown up as slow queries on the reader DBs, the team had been watching only the readers.

Symptom Service outageDatabase overloadHigh latency

Root cause

Root cause Capacity exhaustion

Component DatabasePostgreSQL vacuum
Contributing factor Inefficient queryMonitoring blind spotExcessive blast radius
Trigger
A 100% discount event on 125 courses, from 3 to 17 January and 18 to 24 January 2022, went viral and brought 5 to 20 times the usual traffic.

Detection

The team followed the outages through hourly transaction counts, RDS metrics, monitoring and slow query alerts.

Remediation

Action items

Lessons

Because every database bottleneck appeared as slow queries on the reader DBs, watching only the readers hid the root cause on the writer. In PostgreSQL, frequent updates to a table with large columns produce large dead tuples in bulk and slow every query on that table. Before the second event the team could not run a meaningful performance test because the first week showed that almost any page could fail under load. The event exposed high sensitivity to failure in every area, one table's failure spreading to the whole service, hard-to-tune table and cache structures, unintuitive SQL code, code without types or tests, no stable batch environment, and many hidden failure points not yet found.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Database, PostgreSQL vacuum among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.