Incident Library

Organisation

Honeycomb

Honeycomb · honeycomb

2 cases · 2023 to 2024

Root cause Capacity exhaustion1Breaking change in an upgrade1
Component In-memory cache1Cloud load balancer1Application database1Ingress controller1
Contributing factor Automation amplification1Change with a delayed effect1Monitoring blind spot1Retry storm1Unrepresentative staging1
Symptom Service outage2Container crash loop1Database overload1High error rate1High latency1Stale endpoint routing1

Cases

  1. IL-0203 A routine Kubernetes cluster system update made the AWS Load Balancer Controller fail, so new pods were never registered with the ALB and Honeycomb's EU1 ingest and UI went down once the last old pods were replacedHoneycomb · 2024

    Root cause Breaking change in an upgrade

  2. IL-0202 A burst of queries and database writes exhausted connection pools and memory on Honeycomb's ingest hosts, and Kubernetes crash-loop back-off kept failed hosts offline so ingest capacity stayed too low to recoverHoneycomb · 2023 · sev1

    Root cause Capacity exhaustion