Incident Library

Organisation

Clever Cloud

Clever Cloud · clever-cloud

1 case · 2024

Root cause Configuration change
Component Automated capacity managerLinux kernelMessage brokerTelemetry agent
Contributing factor Automation amplificationCircular dependencyUnreviewed defaultsMonitoring blind spotNo configuration testsRetry stormIneffective rollback
Symptom Autoscaler misbehaviourDeploys failingMemory pressureMonitoring outageSustained high node CPUNode kernel panicOOMKilled

Cases

  1. IL-0285 Clever Cloud's Pulsar brokers stalled in a minor upgrade, then hypervisors overloaded and crashed with kernel panicsClever Cloud · 2024

    Root cause Configuration change