Incident Library

IL-0488 Kit · 2022-12-24databaseaws

Kit's RDS MySQL write latency rose 100x and running threads jumped from 9 to 900, backing up Sidekiq job queues

Kit runs MySQL on Amazon RDS in High Availability (Multi-AZ) in us-east-2; as of July 2023 the database was 13.75 TB on a db.r6g.16xlarge instance. In Multi-AZ, replication to the standby is synchronous, so every transaction commits on the standby as well as the active instance. After an EBS interruption on the standby, AWS ran a replaceSecondaryHostAndVolumesTask: a snapshot of the primary was restored onto new secondary volumes. Those volumes had a "first touch penalty", and while the secondary synced with the primary, write I/O latency rose from 1-2 ms to 140 ms around 09:30 UTC. AWS recorded WriteLatency as raised between 09:34 and 14:56 UTC.

Slow writes held up Kit's Sidekiq jobs, queues grew and Redis memory filled. The manual failover requested at about 13:30 UTC timed out, because a failover does not happen while the secondary is not in sync, and the secondary was not in sync until 14:59:35 UTC. At about 14:30 UTC Kit scaled the RDS storage by 10%, from 12,500 GB to 13,750 GB, which took the instance through the "optimizing storage" state in which AWS moves EBS volumes. By about 15:05 UTC write latency was back to 1-2 ms; jobs drained, Redis memory returned to safe levels, and the service had stayed up throughout, only slower.

Severity
–
Scope
Kit's RDS MySQL primary and the Sidekiq job processing that depends on it slowed for several hours; the service never stopped
Users affected
–
Region
us-east-2 (use2-az1)
Time to detect
46m
Time to resolve
5h 22m

Symptoms

At 5:20 am EST on December 24, 2022, Kit's infrastructure team got its first page because the number of MySQL threads running on its RDS MySQL primary in us-east-2 had jumped 100x, from 9 to 900, at a time of day when Kit's traffic is very low. Sidekiq queues were backing up.

Kit processes about 5 billion Sidekiq jobs a month, and most of them touch MySQL, so the first suspicion was newly shipped code hurting one queue. The engineer started with the queue that was backed up more than any other and found nothing there. Digging further showed that RDS MySQL write latency had risen more than 100x, from about 1.5 ms to about 150 ms and up to 300 ms, and that slow writes were holding back job processing.

The backlog became urgent because Sidekiq is backed by Redis and limited by Redis memory: jobs were enqueued faster than they were processed, Redis memory was running out quickly, and a Redis crash would have lost the enqueued jobs. At 8:33 am EST the engineer tried to fail the database over to the standby, expecting some downtime, but the failover itself failed and writes stayed in the hundreds of milliseconds.

The night before, at 9:50 pm EST on December 23, an alert from the AWS RSS feed in Slack had reported degraded EC2 instances and EBS volumes in the use2-az1 availability zone, where the MySQL primary runs. The team paid little attention to it, because hosts in trouble can be replaced and AWS marked the event resolved after 24 minutes. Only later did the team connect the MySQL slowdown to that EBS degradation.

Symptom Database overloadThroughput degradation

Root cause

Root cause Cloud provider outage

Component DatabaseCloud block storageDatabase replication
Contributing factor Ignored early signal
Trigger
An EBS interruption on the standby of the Multi-AZ RDS instance, reported by AWS as an LSE at 02:12:47 UTC on 2022-12-24, started a replaceSecondaryHostAndVolumesTask that replaced the secondary host and volumes.

Detection

A page fired at 5:20 am EST because MySQL threads running jumped from 9 to 900. An AWS RSS feed alert in Slack about degraded EC2 and EBS in use2-az1 had arrived the evening before and was not acted on.

Remediation

Action items

–

Lessons

Multi-AZ RDS keeps data consistent and durable, but synchronous replication means each write also waits for the standby, so a degraded EBS volume on the standby while it is being replaced slows writes on the primary. Large databases recover more slowly from such events than small ones. A single-AZ instance would not have been hit by a standby problem, but would have suffered hours of downtime if its own zone failed, and that trade-off has to be weighed when deploying production RDS instances.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Database, Cloud block storage, Database replication among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.