Incident Library

Organisation

loveholidays

loveholidays · loveholidays

3 cases · 2019 to 2020

Root cause Capacity exhaustion2Cloud provider outage1
Component Node pool3Cluster autoscaler2conntrack table1GKE control plane1HAProxy1Horizontal pod autoscaler1Ingress controller1kube-proxy1VPC subnet or pod address range1
Contributing factor Unknown limit2Unreviewed defaults1Ignored early signal1No alert on limit1No capacity planning1Overlapping failures1Unrepresentative staging1
Symptom Scale-up failure2Pods Pending2Autoscaler misbehaviour1Connection resets1High error rate1Intermittent connection failures1IP address exhaustion1Service outage1Upgrade stall1

Cases

  1. IL-0013 The conntrack table filled up on downsized GKE nodes and a high-traffic aggregation proxy saw climbing connection resetsloveholidays · 2020

    Root cause Capacity exhaustion

  2. IL-0015 A GKE cluster's /16 pod address range ran out at 256 nodes, leaving new pods Pending and node scale-ups failingloveholidays · 2020

    Root cause Capacity exhaustion

  3. IL-0014 A GKE upgrade to a withdrawn version stalled on slow drains and a stuck node pool, then broke an ingress and confused the HPAloveholidays · 2019

    Root cause Cloud provider outage