Incident Library
Applied filters ComponentJobs and CronJobsSymptomHigh error rateFactorUnreviewed defaultsClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0435 Error rates rose as Aurora MySQL killed connections and declined queries to avoid OOM while load metrics stayed flat
    CyberAgent2026
    Symptoms

    An alert fired one day because an application's error rate had suddenly risen. The error log of the Aurora MySQL RDS instance for that time showed Aurora killing connections and declining queries to avoid running out of memory. The killed queries also made application latency worse.

    High error rateMemory pressure
    Trigger

    An application release added a batch job run every minute by a Kubernetes CronJob. Each run's short-lived pod connected to the database from a new pod IP.

    Root cause
    Capacity exhaustion
    Component
    DatabaseJobs and CronJobs
    Contributing factor
    Unreviewed defaultsChange with a delayed effect
    Detection

    An alert fired on the application's error rate.

    Remediation
    • Set performance_schema to manual management and capped two related Performance Schema parameters, unlimited by default, at 100 in the parameter group, then restarted the RDS instance
    Lessons

    When system alerts fire but the database's load metrics look calm, Performance Schema growth is worth suspecting. The two parameters that were capped are unlimited by default, so connections from ever-changing source addresses keep adding to the records. Capping them has a cost: once the cap is exceeded, per-host figures in Performance Insights are missing. Putting RDS Proxy between the application and Aurora MySQL to keep source hosts stable is another possible measure.

    View case
Check your cluster for this

This case involves Database and Jobs and CronJobs. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion1 case

Component
DatabaseJobs and CronJobs
Contributing factor
Unreviewed defaultsChange with a delayed effect
Organisation
CyberAgent
  1. IL-0435 Error rates rose as Aurora MySQL killed connections and declined queries to avoid OOM while load metrics stayed flat
    CyberAgent2026