Cases2 cases

1 to 2 of 2

1. IL-0435 Error rates rose as Aurora MySQL killed connections and declined queries to avoid OOM while load metrics stayed flat

   CyberAgent2026

   Symptoms

   An alert fired one day because an application's error rate had suddenly risen. The error log of the Aurora MySQL RDS instance for that time showed Aurora killing connections and declining queries to avoid running out of memory. The killed queries also made application latency worse.

   High error rateMemory pressure

   Trigger

   An application release added a batch job run every minute by a Kubernetes CronJob. Each run's short-lived pod connected to the database from a new pod IP.

   Root cause

   Capacity exhaustion

   Component

   DatabaseJobs and CronJobs

   Contributing factor

   Unreviewed defaultsChange with a delayed effect

   Detection

   An alert fired on the application's error rate.

   Remediation

   • Set performance_schema to manual management and capped two related Performance Schema parameters, unlimited by default, at 100 in the parameter group, then restarted the RDS instance

   Lessons

   When system alerts fire but the database's load metrics look calm, Performance Schema growth is worth suspecting. The two parameters that were capped are unlimited by default, so connections from ever-changing source addresses keep adding to the records. Capping them has a cost: once the cap is exceeded, per-host figures in Performance Insights are missing. Putting RDS Proxy between the application and Aurora MySQL to keep source hosts stable is another possible measure.

   View case
2. IL-0232 Calico pods in the codfw WikiKube cluster hit their resource limits and crash-looped, taking nodes out one by one

   Wikimedia Foundation2024

   Symptoms

   At 05:01 UTC on 9 October 2024 a page fired for GatewayBackendErrorsHigh, elevated 5xx errors from page-analytics_cluster in codfw, followed by a growing number of KubernetesCalicoDown alerts for WikiKube nodes not running the calico-node pod and by MediaWikiMemcachedHighErrorRate. For about an hour uncached MediaWiki requests, such as those of logged-in users, edits and API calls, saw high latency or errors.

   High error rateHigh latencyContainer crash loopProbe failuresControl plane lag

   Trigger

   An operator used the new mwscript-k8s utility to run a batch of video transcode maintenance, the TimedMediaHandler requeueTranscodes.php script. mwscript-k8s deploys a whole helm release for every script invocation, so the batch created close to 2k Calico NetworkPolicies along with many other resources over several hours.

   Root cause

   Capacity exhaustion

   Component

   CalicoHelmJobs and CronJobs

   Contributing factor

   Change with a delayed effectNo runbook

   Detection

   Automated monitoring detected the issue and paged the on-call SRE: GatewayBackendErrorsHigh at 05:01:36 UTC, 16 KubernetesCalicoDown alerts at 05:03:32 UTC and MediaWikiMemcachedHighErrorRate at 05:04:15 UTC.

   Remediation

   • Raised the resource limits of the Calico and Typha pods and applied the change with helmfile, which redeployed all Calico pods

   Action items

   • Stop mwscript-k8s from creating so many resources

   Lessons

   The incident came close to a full outage. Automated monitoring caught it and many responders found the cause, but end users were affected and responders were unsure whether they could switch over to eqiad. The problem was noticed just before the hand-off from Americas to EMEA SRE shifts, a period with a long-known lack of coverage, and it helped that the transcodes were not queued on a Friday evening in the US.

   View case

By root cause1