Incident Library

Organisation

Wealthy

Wealthy · wealthy

1 case · 2025

Root cause Breaking change in an upgrade
Component ConfigMaps and SecretsClient-side DNS resolver and cacheGKE control planeNode poolPersistent volumes and stateful sets
Contributing factor No headroomIneffective rollbackSingle point of failureDeferred upgrades
Symptom Scale-up failureContainer crash loopDNS resolution failuresService outage

Cases

  1. IL-0101 After Google forced a GKE control plane upgrade to 1.33.2 at a deprecation deadline, no new node could initialise, so autoscaling and spot node replacement stopped for 15 hoursWealthy · 2025 · sev1

    Root cause Breaking change in an upgrade