Cases2 cases

1 to 2 of 2

1. IL-0447 An every RDS for MySQL instance slowed badly when its gp2 EBS BurstBalance ran out and IOPS fell to the baseline

   every2024

   Symptoms

   The BurstBalance of an RDS instance began to fall rapidly and finally ran out, and the database's performance dropped sharply. The instance ran MySQL 5.7.44 on db.t3.medium with general purpose SSD (gp2) storage of 30GiB.

   Database overloadThroughput degradation

   Trigger

   A complex SELECT query began writing large amounts of data to temporary tables, which raised WriteIOPS and used up the burst credits.

   Root cause

   Capacity exhaustion

   Component

   DatabaseCloud block storage

   Contributing factor

   Inefficient query

   Action items

   • Consider tuning the query by reviewing indexes and rewriting it
   • Consider moving the storage from gp2 to gp3 or increasing its size to raise the baseline IOPS
   • Consider adjusting tmp_table_size and max_heap_table_size
   • Consider moving to a more powerful instance type
   • Consider monitoring BurstBalance with CloudWatch alerts and adding external monitoring such as Pingdom

   Lessons

   A complex query can have a large effect on I/O, so database design and queries need continuous improvement, and query performance needs regular monitoring.

   View case
2. IL-0401 Prow jobs on the k8s-prow-build GKE cluster failed and merges stalled as nodes hit IOPS throttling and kernel panics

   Kubernetes project2019

   Symptoms

   On 30 May 2019, in the run-up to the Kubernetes 1.15 code freeze, Prow test jobs in the k8s-prow-build cluster failed in surprising ways or were aborted partway through. Jobs were reported to GitHub as failed while Deck still showed them as running, because pods exited without uploading `finished.json`, the file most Prow tooling uses to tell a job's state.

   Node NotReadyNode kernel panicPods stuck in TerminatingThroughput degradation

   Trigger

   A spike in test load before the Kubernetes 1.15 code freeze pushed disk I/O on the build nodes far beyond the IOPS that GCP allows for 250 GB disks.

   Root cause

   Capacity exhaustion

   Component

   Cloud block storageNode root diskkubeletCI system

   Contributing factor

   Monitoring blind spotIgnored early signal

   Detection

   Several users reported jobs that GitHub showed as failed but Deck showed as still running, and the first report came on Slack at 13:26. Alerting for these failures did not exist or did not work.

   Remediation

   • Requested and received additional disk quota
   • Brought up 55 nodes with 500 GB disks in a new node pool
   • Migrated all nodes from the 250 GB disk pool to the 500 GB disk pool by draining old nodes, finishing at 23:55

   Action items

   • Create or fix monitoring so the oncall is paged when tide stops doing things
   • Set a process and guidelines for adding new members to the oncall rotation, with preparation and extra support

   Lessons

   Prow metrics were not recorded before the incident, which made it hard to understand, and alerting for these failures did not exist or did not work. After the cause was found and a fix devised, nothing happened for another two hours. The team counts it as luck that someone who recognised IOPS as the problem happened to be around to look.

   View case

By root cause1