Incident Library

IL-0444 Repro · 2024-10-23 (published)databaseaws

Reader connections to Repro's Aurora MySQL cluster were cut with Lost connection errors every time ALTER TABLE ran

Every schema change on Repro's Aurora MySQL cluster caused application errors from the time the cluster was on Aurora MySQL 5.7: an ALTER TABLE on the writer cut the reader connections that were reading the altered table, and the clients saw `Lost connection to MySQL server during query`. The development cluster had no reader, so the failure could not be reproduced until one was added. AWS Support explained that this is documented behaviour: DDL on the primary interrupts Aurora Replica connections that are using the changed object.

Repro made the applications retry once after a connection error. For the Rails applications, a monkey patch module ActiveRecord::RetryOnLostConnection prepended to ActiveRecord::ConnectionAdapters::AbstractMysqlAdapter catches ActiveRecord::StatementInvalid whose message includes "Lost connection to MySQL server", logs a warning, reconnects and retries the query one time. It does not retry inside an open transaction, because a retry in a new reader session could return a different result than the earlier reads of the same transaction, and it does not retry on the writer connection. The patch has since been refined by other engineers.

Repro also mentions a separate problem with the same trigger: during an ALTER TABLE, a reader can briefly fail to see the altered table and return "Error 1146: Table '...' doesn't exist". That one is a bug, fixed in Aurora MySQL 3.05.0.

Severity
–
Scope
Application processes that read a table through an Aurora reader while ALTER TABLE ran on it got connection errors
Users affected
–
Region
–
Time to detect
–
Time to resolve
–

Symptoms

After Repro moved to Aurora MySQL 5.7, running an ALTER TABLE in production, even one that finished in under a second, made the processes reading that table fail with `Lost connection to MySQL server during query`. MySQL client connections were cut right after the ALTER TABLE ran, and many application errors followed. By 8 January 2021 this had happened three times in a row.

The failure did not reproduce in the development environment, because the development Aurora MySQL cluster had no reader. After a reader was added there, it reproduced: right after an ALTER TABLE ran on a cluster with a reader, MySQL client connections that were sending SQL referring to the altered table to the reader were closed, with a FIN coming back.

The team first thought this was a bug and sent the reproduction steps to AWS Support. AWS Support pointed to documented Aurora behaviour: DDL statements run on the primary instance can interrupt connections on the Aurora Replicas, and a replica connection that is actively using a table is interrupted when that table is changed by DDL on the primary.

Symptom Connection resetsHigh error rate

Root cause

Root cause Breaking change in an upgrade

Component Database
Contributing factor Unrepresentative staging
Trigger
Running ALTER TABLE on the Aurora MySQL writer while readers were running queries that used the same table.

Detection

–

Remediation

Action items

–

Lessons

–

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Database among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.