Incident Library

Organisation

Moonlight

Moonlight · moonlight

1 case · 2019

Root cause Capacity exhaustion
Component Application databaseLinux kernelPod scheduling and placement
Contributing factor Unverified assumptionAutomation amplificationSingle point of failure
Symptom High error rateIntermittent connection failuresSustained high node CPUNode NotReadyService outage

Cases

  1. IL-0064 CPU-heavy pods packed onto one Google Cloud node drove it to kernel panics, and the crash moved from node to node until the site was downMoonlight · 2019

    Root cause Capacity exhaustion