Cases2 cases

1 to 2 of 2

1. IL-0435 Error rates rose as Aurora MySQL killed connections and declined queries to avoid OOM while load metrics stayed flat

   CyberAgent2026

   Symptoms

   An alert fired one day because an application's error rate had suddenly risen. The error log of the Aurora MySQL RDS instance for that time showed Aurora killing connections and declining queries to avoid running out of memory. The killed queries also made application latency worse.

   High error rateMemory pressure

   Trigger

   An application release added a batch job run every minute by a Kubernetes CronJob. Each run's short-lived pod connected to the database from a new pod IP.

   Root cause

   Capacity exhaustion

   Component

   DatabaseJobs and CronJobs

   Contributing factor

   Unreviewed defaultsChange with a delayed effect

   Detection

   An alert fired on the application's error rate.

   Remediation

   • Set performance_schema to manual management and capped two related Performance Schema parameters, unlimited by default, at 100 in the parameter group, then restarted the RDS instance

   Lessons

   When system alerts fire but the database's load metrics look calm, Performance Schema growth is worth suspecting. The two parameters that were capped are unlimited by default, so connections from ever-changing source addresses keep adding to the records. Capping them has a cost: once the cap is exceeded, per-host figures in Performance Insights are missing. Putting RDS Proxy between the application and Aurora MySQL to keep source hosts stable is another possible measure.

   View case
2. IL-0373 Heroku Postgres Essential-tier databases in the EU region ran out of memory and refused connections, twice in four days

   Heroku2024

   Symptoms

   From 22:31 UTC on 27 July 2024, 33 Heroku Postgres clusters in the EU (Dublin) region could not serve requests and ran into memory problems, and some Essential-tier customers could not connect to their databases. Customers trying to reach an affected database received out of memory errors.

   Memory pressureService outage

   Trigger

   A third-party vendor deployed a change to a monitoring agent that runs on Heroku's Postgres database clusters.

   Root cause

   Software defect

   Component

   DatabaseTelemetry agent

   Contributing factor

   Monitoring blind spot

   Detection

   Heroku Support paged the technology team at 07:50 UTC on 28 July based on customer support tickets. The second occurrence also reached the technology team as an escalation from Heroku Support.

   Remediation

   • Allocated more resources and restarted the Postgres clusters, which fixed the first outage temporarily
   • Provisioned the previous version of the third-party monitoring software, which did not leak memory
   • The vendor applied an interim mitigation

   Action items

   • Add enhanced metrics collection for the third-party monitoring software
   • Evaluate the vendor's fixed release of the monitoring software and roll it out if acceptable

   View case

By root cause2