Incident Library

Organisation

Yonghyun Kim

Yonghyun Kim · yonghyun-kim

2 cases · 2025

Root cause Dependency failure1Faulty deployment1
Component Cloud load balancer1Liveness and readiness probes1Ingress controller1Pod termination1
Contributing factor Unreviewed defaults2Unverified assumption1Automation amplification1Excessive blast radius1Monitoring blind spot1
Symptom Service outage2Container crash loop1High error rate1Probe failures1

Cases

  1. IL-0188 Rolling updates on EKS returned 502 errors for about 30 seconds because Kubernetes deleted old pods as soon as new pods passed readiness, before the ALB had marked the new pod targets healthyYonghyun Kim · 2025

    Root cause Faulty deployment

  2. IL-0187 A brief Spring Cloud Config Server downtime failed every client application's liveness probe, because the actuator health check included the config server, and Kubernetes restarted the pods in a kill loopYonghyun Kim · 2025

    Root cause Dependency failure